Chapter 3: Network requests

3.1 Fetch

JavaScript can send network requests to the server and load new information whenever it’s needed.

AJAX: Asynchronous JavaScript And XML

let promise = fetch(url, [options])

• url

• options: method, headers, etc.

• The promise resolves with an object of the built-in Response class as soon as the server responds with headers. The promise rejects if the fetch was unable to make HTTP-request.

• status: HTTP status code

• ok: boolean, true if code is 200-299

• Response provides multiple promise-based methods to access the body in various.

• response.text()

• response.json()

• response.formData()

• response.blob()

• response.arrayBuffer()

• response.body: ReadableStream object

We can choose only one body-reading method.

Headers

let response = fetch(protectedUrl, {
  headers: {
    Authentication: 'secret'
  }
});

response.headers.get('Content-Type'));

There’s a list of forbidden HTTP headers that we can’t set: Accept-Encoding, Connection, Content-Length, Keep-Alive, etc.

POST requests

• method: POST

• body: one of the following:

• • string

• • FormData

• • Blob/BufferSource

• • URLSearchParams

let response = await fetch('/article/fetch/post/user', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json;charset=utf-8'
  },
  body: JSON.stringify(user)
});

3.2 FormData

let formData = new FormData([form]);

If HTML form element is provided, it automatically captures its fields. It has a special type: Content-Type: multipart/form-data.

• formData.append(name, value): add a form field with the given name and value.

• formData.append(name, blob, fileName): add a file field

• formData.delete(name)

• formData.get(name)

• formData.has(name)

• formData.set(name, value): remove other name and add the new one (guaranteed unique)

3.3 Fetch: Download progress

The fetch method allows to track download progress.

response.body gives full control over the reading process, and we can count how much is consumed at any moment.

const reader = response.body.getReader();

while(true) {
  const {done, value} = await reader.read();
  if (done) {
    break;
  }
  console.log(`Received ${value.length} bytes`)
}

3.4 Fetch: Abort

AbortController can be used to abort not only fetch, but other asynchronous tasks as well.

1. Create a controller.

let controller = new AbortController();

It has a single method abort(), and a single property signal.

When abort() is called:

• abort event triggers on controller.signal.

• controller.signal.aborted property becomes true.

• Pass the signal property to fetch option.

fetch(url, {
  signal: controller.signal
});

1. To abort, call controller.abort(). Its promise rejects with an error AbortError.

AbortController is scalable, it allows to cancel multiple fetches at once.

3.5 Fetch: Cross-Origin Requests

If we send a fetch request to another web-site, it will probably fail.

Cross-origin requests require special headers from the remote side.

Simple requests

A simple request is a request that satisfies two conditions: 1. GET, POST or HEAD 2. Headers: Accept, Accept-Language, Content-Language, Content-Type (application/x-www-form-urlencoded, multipart/form-data, or text/plain)

A “simple request” can be made with a <form> or a <script>, without any special methods.

CORS for simple requests

Request

GET /request
Host: anywhere.com
Origin: https://javascript.info
...

Response

200 OK
Content-Type:text/html; charset=UTF-8
Access-Control-Allow-Origin: https://javascript.info
...

If the Access-Control-Allow-Origin contains the origin or equals *, the response is successful, otherwise an error.

Because Referer is unreliable (can be modified or removed by fetch), Origin was invented, and the correctness of that header is ensured by the browser.

Response headers

For cross-origin request, by default JavaScript may only access simple response headers:

• Cache-Control

• Content-Language

• Content-Type

• Expires

• Last-Modified

• Pragma

To grant JavaScript access to any other response header, the server must send Access-Control-Expose-Headers header.

Access-Control-Allow-Origin: https://javascript.info
Access-Control-Expose-Headers: Content-Length,API-Key

Non-simple requests

A preflight request uses method OPTIONS, no body and two headers:

• Access-Control-Request-Method: the method of the non-simple request.

• Access-Control-Request-Headers: a comma-separated list of its non-simple HTTP-headers.

If the server agrees to serve the requests, then it should respond with empty body, status 200 and headers:

• Access-Control-Allow-Method

• Access-Control-Allow-Headers

• Access-Control-Max-Age: a number of seconds to cache the permissions.

Then the actual request is sent, the previous simple scheme is applied.

Credentials

A cross-origin request by default does not bring any credentials (cookies or HTTP authentication).

Request

fetch('http://another.com', {
  credentials: "include"
});

Response

Access-Control-Allow-Credentials: true

Access-Control-Allow-Origin is prohibited from using * for requests with credentials.

3.6 Fetch API

let promise = fetch(url, {
  method: "GET", // POST, PUT, DELETE, etc.
  headers: {
    // the content type header value is usually auto-set
    // depending on the request body
    "Content-Type": "text/plain;charset=UTF-8"
  },
  body: undefined // string, FormData, Blob, BufferSource, or URLSearchParams
  referrer: "about:client", // or "" to send no Referer header,
  // or an url from the current origin
  referrerPolicy: "no-referrer-when-downgrade", // no-referrer, origin, same-origin...
  mode: "cors", // same-origin, no-cors
  credentials: "same-origin", // omit, include
  cache: "default", // no-store, reload, no-cache, force-cache, or only-if-cached
  redirect: "follow", // manual, error
  integrity: "", // a hash, like "sha256-abcdef1234567890"
  keepalive: false, // true
  signal: undefined, // AbortController to abort request
  window: window // null
});

referrer, referrerPolicy

The referrer option allows to set any Referer within the current origin or remove it.

fetch('/page', {
  referrer: "" // no Referer header
});

The referrerPolicy option sets general rules for Referer.

• "no-referrer-when-downgrade": Do not send Referer when we send a request from HTTPS to HTTP.

• no-referrer

• origin

• origin-when-cross-origin: Send origin as Referer when we send a cross-origin request.

• same-origin: Send full Referer to the same origin, but no Referer for cross-origin requests.

• strict-origin: Send origin as Referer, but don’t send it for downgrade requests.

• strict-origin-when-cross-origin: For same-origin send full Referer, for cross-origin send only origin, but don’t send it for downgrade requests.

• unsafe-url: Always send full url in Referer.

mode

The mode option is a safe-guard that prevents occasional cross-origin requests:

• cors: the default, cross-origin requests are allowed

• same-origin: cross-origin requests are forbidden

• no-cors

credentials

The credentials option specifies whether fetch should send cookies and HTTP-Authorization headers with the request.

• same-origin: the default, don’t send for cross-origin requests

• include: always send.

• omit: never send.

cache

By default, fetch requests make use of standard HTTP-caching. It uses Expires, Cache-Control headers, sends If-Modified-Since, etc.

• default

• no-store

• reload

• no-cache

• force-cache

• only-if-cached

redirect

• follow

• error: throw error in case of HTTP-redirect.

• manual: don’t follow HTTP-redirect, but response.url will be the new URL, and response.redirected will be true.

integrity

The integrity option allows to check if the response matches the known-ahead checksum. (SHA-256, SHA-384, and SHA-512)

fetch('http://site.com/file', {
  integrity: 'sha256-abcdef'
});

keepalive

The keepalive option indicates that the request may outlive the webpage that initiated it.

Normally, when a document is unloaded, all associated network requests are aborted. But keepalive option tells the browser to perform the request in background, even after it leaves the page.

• The body limit for keepalive requests is 64kb.

• We can’t handle the response if the document is unloaded.

3.7 URL objects

Creating a URL

let url = new URL(url, [base]);

let jsInfoUrl = new URL('/profile/admin', 'https://javascript.info');

• url: full URL or only path.

• base: if set and url argument has only path, then the full URL is generated.

let url = new URL('https://javascript.info/url');

alert(url.protocol); // https:
alert(url.host);     // javascript.info
alert(url.pathname); // /url

• href: the full url, same as url.toString().

• protocol: example: https

• search: a string of parameters, starts with the question mark ?.

• hash: starts with the hash character #.

SearchParams

new URL('https://google.com/search?query=JavaScript')

Parameters need to be encoded if they contain spaces, non-latin letters, etc.

• append(name, value)

• delete(name)

• get(name)

• getAll(name)

• has(name)

• set(name, value)

• sort()

Encoding

Encoding strings

• encodeURI: only characters that are totally forbidden in URL.

• decodeURI

• encodeURIComponent: forbidden characters and #, $, &, +, ,, /, :, ;, =, ?, @.

• decodeURIComponent

let music = encodeURIComponent('Rock&Roll');

let url = `https://google.com/search?q=${music}`;
alert(url); // https://google.com/search?q=Rock%26Roll

3.8 XMLHttpRequest

In modern web-development XMLHttpRequest is used for three reasons:

• Historical reasons

• Support old browsers without polyfills

• Features that are not supported by fetch, such as tracking upload progress

The basics

1. Create XMLHttpRequest:

let xhr = new XMLHttpRequest();

1. Initialize it:

xhr.open(method, URL, [async, user, password])

• method: HTTP-method

• URL

• async

• user, password

• Send it out:

xhr.send([body])

1. Listen to xhr events for response.

2. load: triggers when the request is complete (even if HTTP status is like 400 or 500) and the response is downloaded.

3. error

4. progress: triggers periodically while the response is being downloaded, reports how much has been downloaded.

xhr.onload = function() {
  alert(`Loaded: ${xhr.status} ${xhr.response}`);
};

xhr.onprogress = function(event) {
  // triggers periodically
  // event.loaded: how many bytes downloaded
  // event.lengthComputable: true if the server sent Content-Length header
  // event.total: total number of bytes (if lengthComputable)
  alert(`Received ${event.loaded} of ${event.total}`);
};

Once the server has responded, xhr has these properties:

• status

• statusText: example: OK, NotFound, Forbidden

• response

We can also specify a timeout using the corresponding property:

xhr.timeout = 10000;

xhr.responseType

• "": string

• text: string

• arraybuffer

• blob

• document: XML

• json

xhr.open('GET', 'example.json');

xhr.responseType = 'json';

xhr.onload = function() {
  let responseObj = xhr.response;
  alert(responseObj.message);
};

xhr.readyState

UNSENT = 0; // initial state
OPENED = 1; // open called
HEADERS_RECEIVED = 2; // response headers received
LOADING = 3; // response is loading (a data packed is received)
DONE = 4; // request complete

We can track them using readystatechange event. State 3 repeats every time a data packet is received over the network.

To terminate the request, call xhr.abort().

Synchronous requests

If in the open method the third parameter async is set to false, the request is made synchronously. JavaScript execution pauses at send() and resumes when the response is received.

However, if a synchronous call takes too much time, the browser may suggest to close the “hanging” webpage.

HTTP-headers

setRequestHeader(name, value)

xhr.setRequestHeader('Content-Type', 'application/json');

Several headers are managed exclusively by the browser, e.g. Referer and Host. It can’t undo setRequestHeader.

xhr.setRequestHeader('X-Auth', '123');
xhr.setRequestHeader('X-Auth', '456');

// the header will be:
// X-Auth: 123, 456

getResponseHeader(name)
xhr.getResponseHeader('Content-Type')

getAllResponseHeaders()

Cache-Control: max-age=31536000
Content-Length: 4260
Content-Type: image/png
Date: Sat, 08 Sep 2012 16:53:16 GMT

POST, FormData

To make a POST request, we can use the built-in FormData object, which is sent with multipart/form-data encoding.

let formData = new FormData([form]); // creates an object, optionally fill from <form>
formData.append(name, value); // appends a field

Upload progress

The progress event triggers only on the downloading stage. We can use xhr.upload to track upload events.

It generates these events:

• loadstart – upload started.

• progress – triggers periodically during the upload.

• abort – upload aborted.

• error – non-HTTP error.

• load – upload finished successfully.

• timeout – upload timed out (if timeout property is set).

• loadend – upload finished with either success or error.

xhr.upload.onprogress = function(event) {
  alert(`Uploaded ${event.loaded} of ${event.total} bytes`);
};

3.9 Resumable file upload

Not-so-useful progress event

The event triggers when the data is sent, instead of receving by the server. Maybe it was buffered by a local network proxy, or maybe the remote server process just died and couldn’t process them.

Algorithm

1. Create an id to uniquely identify the file.

2. Send a request to the server, asking how many bytes it already has.

3. Use Blob method slice to send the file from the break point.

3.10 Long polling

Long polling is the simplest way of having persistent connection with server, that doesn’t use any specific protocol like WebSocket or Server Side Events.

Regular polling

The simplest way to get new information from the server is periodic polling.

• Messages are passed with a delay up to the period.

• The server may have too many requests to handle.

Long pooling

Long polling works great in situations when messages are rare.

Every message is a separate request, supplied with headers and authentication overhead.

1. A request is sent to the server.

2. The server doesn’t close the connection until it has a message to send.

3. When a message appears – the server responds to the request with it.

4. The browser makes a new request immediately.

async function subscribe() {
  let response = await fetch("/subscribe");

  if (response.status == 502) {
    await subscribe();
  } else if (response.status != 200) {
    showMessage(response.statusText);
    await new Promise(resolve => setTimeout(resolve, 1000));
    await subscribe();
  } else {
    let message = await response.text();
    showMessage(message);
    await subscribe();
  }
}

3.11 WebSocket

The WebSocket protocol provides a way to exchange data between browser and server via a persistent connection.

A simple example

let socket = new WebSocket("wss://javascript.info");

Use wss:// protocol is encrypted and more reliable than ws://.

Events

• open – connection established,

• message – data received,

• error – websocket error,

• close – connection closed.

Opening a websocket

When new WebSocket(url) is created, it starts connecting immediately.

Here’s an example of browser headers for request:

GET /chat
Host: javascript.info
Origin: https://javascript.info
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: Iv8io/9s+lYFgZWcXczP8Q==
Sec-WebSocket-Version: 13

• Origin: It allows the server to decide whether or not to talk WebSocket with this website.

• Connection: Upgrade: The signal to change to protocol.

• Upgrade: websocket: The requested protocol.

• Sec-WebSocket-Key: Random security key.

• Sec-WebSocket-Version: WebSocket protocol version. (Current: 13)

The server should send code 101 response:

101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: hsBlbuDTkk24srzEOTBUlZAlC2g=

Then the connection is made. WebSocket handshake can’t be emulated.

Extensions and subprotocols

The additional headers Sec-WebSocket-Extensions and Sec-WebSocket-Protocol describe extensions and subprotocols.

• Sec-WebSocket-Extensions: deflate-frame: the browser supports data compression.

• Sec-WebSocket-Protocol: soap, wamp: we’d like to transfer the data in SOAP or WAMP ("The WebSocket Application Messaging Protocol") protocols.

let socket = new WebSocket("wss://javascript.info/chat", ["soap", "wamp"]);

The server should respond with a list of protocols and extensions that it agrees to use.

Data transfer

WebSocket communication consists of “frames” – data fragments, that can be sent from either side, and can be of several kinds:

• text frames

• binary data frames

• ping/pong frames: used to check the connection (sent by server and responded automatically by browser)

• connection close frame

WebSocket .send() method can send either text or binary data.

When we receive the data, text always comes as string. And for binary data, we can choose between Blob and ArrayBuffer formats.

Rate limiting

The socket.bufferedAmount property stores how many bytes are buffered at this moment, waiting to be sent over the network.

Connection close

socket.close([code], [reason]);

• code: a special WebSocket closing code.

• reason: a string that describes the reason of closing.

Most common codes:

• 1000: the default closure

• 1006: connection was lost (can't be sent manually)

• 1001: the party is going away (server shuts down or user leaves the page)

• 1009: the message is too big to process

• 1011: unexpected error on server

Connection state

There's the socket.readyState property with values:

• 0: CONNECTING

• 1: OPEN

• 2: CLOSING

• 3: CLOSED

3.12 Server Sent Events

The Server-Sent Events specification describes a built-in class EventSource, that keeps connection with the server and allows to receive events from it.

It's simpler than WebSocket:

• Only server sends data

• Only text

• Regular HTTP protocol

• Auto-reconnect

Getting messages

To start receiving messages, we just need to create new EventSource(url).

The server should respond with status 200 and the header Content-Type: text/event-stream.

data: Message 1

data: Message 2

data: Message 3

• A message text goes after data:, the space after the colon is optional.

• Messages are delimited with double line breaks \n\n.

• To send a line break \n, we can immediately send one more data.

eventSource.onmessage = function(event) {
  console.log("New message", event.data);
};

Cross-origin requests

EventSource supports cross-origin requests. The remote server will get the Origin header and must respond with Access-Control-Allow-Origin to proceed.

let source = new EventSource("https://another-site.com/events", {
  withCredentials: true
});

Reconnection

There’s a small delay between reconnections, a few seconds by default. The server can set the recommended delay using retry: in response.

retry: 15000

• If the browser knows that there’s no network connection at the moment, it may wait until the connection appears, and then retry.

• If the server wants the browser to stop reconnecting, it should respond with HTTP status 204.

  If the browser wants to close the connection, it should call eventSource.close().

Message id

When a connection breaks due to network problems, either side can’t be sure which messages were received. To correctly resume the connection, each message should have an id field.

data: Message 1
id: 1

When a message with id: is received, the browser:

• Sets the property eventSource.lastEventIdto its value.

• Sends the header Last-Event-ID with that id when reconnecting.

Connection status: readyState

The EventSource object has readyState property.

• 0: Connecting

• 1: Open

• 2: Closed

Event types

• message – a message received, available as event.data.

• open – the connection is open.

• error – the connection could not be established.

The server may specify another type of event with event: ... at the event start.

event: join
data: Bob

To handle custom events, we must use addEventListener:

eventSource.addEventListener('join', event => {
  alert(`Joined ${event.data}`);
});