Initial commit

This commit is contained in:
chinahu-woker
2025-02-16 11:40:12 +08:00
parent d7af560866
commit a151610d25
212 changed files with 14008 additions and 231 deletions
+10
View File
@@ -0,0 +1,10 @@
## 2 July 2019
### [1.0.0](https://github.com/idiocc/text-decoding/compare/v0.0.0-pre...v1.0.0)
- [package] Publish `v1` of the package.
### 0.0.0-pre
- Create `text-decoding` with _[`My New Package`](https://mnpjs.org)_
- [repository]: `src`, [`test`](https://contexttesting.com), [`documentary`](https://readme.page) & [`types`](https://typedef.page).
+27
View File
@@ -0,0 +1,27 @@
MIT License
Copyright (c) 2019 Art Deco Code Limited
+ ES6 code with ECMA modules
+ Zoroaster spec and mask tests
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Original Work by Joshua Bell and contributors
Dual License: Unlicense/Apache 2.0 License
+179
View File
@@ -0,0 +1,179 @@
# text-decoding
[![npm version](https://badge.fury.io/js/text-decoding.svg)](https://npmjs.org/package/text-decoding)
`text-decoding` is a fork of [Polyfill for the Encoding Living Standard's API](https://github.com/inexorabletash/text-encoding) (`text-encoding`) For Node.JS.
This is a polyfill for the [Encoding Living Standard](https://encoding.spec.whatwg.org/) API for the Web, allowing encoding and decoding of textual data to and from Typed Array buffers for binary data in JavaScript.
By default it adheres to the spec and does not support encoding to legacy encodings, only decoding. It is also implemented to match the specification's algorithms, rather than for performance.
```sh
yarn add text-decoding
```
## Table Of Contents
- [Table Of Contents](#table-of-contents)
- [API](#api)
- [`class TextDecoder`](#class-textdecoder)
- [`class TextEncoder`](#class-textencoder)
- [`const EncodingIndexes`](#const-encodingindexes)
- [`getEncoding(label: string): { name: string, labels: Array<string> }`](#getencodinglabel-string--name-string-labels-arraystring-)
- [Encodings](#encodings)
- [Copyright](#copyright)
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/0.svg?sanitize=true"></a></p>
## API
The package is available by importing its named classes and functions:
```js
import { TextEncoder, TextDecoder, EncodingIndexes, getEncoding } from 'text-decoding'
```
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/1.svg?sanitize=true"></a></p>
## `class TextDecoder`
Decodes a Uint8Array into a string.
<table>
<tr><th>Source</th><th>Output</th></tr>
<tr><td>
```js
import { TextDecoder } from 'text-decoding'
const decoded = new TextDecoder('utf-8')
.decode(new Uint8Array([
0x7A, 0xC2, 0xA2, 0xE6, 0xB0, 0xB4, 0xF0,
0x9D, 0x84, 0x9E, 0xF4, 0x8F, 0xBF, 0xBD,
]))
console.log(decoded)
```
</td>
<td>
```
z¢水𝄞􏿽
```
</td></tr>
</table>
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/2.svg?sanitize=true"></a></p>
## `class TextEncoder`
Encodes a string into `Uint8Array` for the given encoding.
As required by the specification, only encoding to utf-8 is supported. If you want to try it out, you can force a non-standard behavior by passing the `NONSTANDARD_allowLegacyEncoding` option to _TextEncoder_ and a label. For example:
```js
import { TextEncoder } from 'text-decoding'
const uint8array = new TextEncoder(
'windows-1252', { NONSTANDARD_allowLegacyEncoding: true })
.encode('hello world')
console.log(uint8array)
```
```js
Uint8Array [ 104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100 ]
```
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/3.svg?sanitize=true"></a></p>
## `const EncodingIndexes`
This is [a map of indexes](src/encoding-indexes.js) used for encoding.
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/4.svg?sanitize=true"></a></p>
## `getEncoding(`<br/>&nbsp;&nbsp;`label: string,`<br/>`): { name: string, labels: Array<string> }`
Returns the normalised name of the encoding and its associated labels.
<table>
<tr><th>Source</th><th>Output</th></tr>
<tr><td>
```js
import { getEncoding } from 'text-decoding'
const encoding = getEncoding('ascii')
console.log(encoding)
```
</td>
<td>
```js
{ labels:
[ 'ansi_x3.4-1968',
'ascii',
'cp1252',
'cp819',
'csisolatin1',
'ibm819',
'iso-8859-1',
'iso-ir-100',
'iso8859-1',
'iso88591',
'iso_8859-1',
'iso_8859-1:1987',
'l1',
'latin1',
'us-ascii',
'windows-1252',
'x-cp1252' ],
name: 'windows-1252' }
```
</td></tr>
</table>
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/5.svg?sanitize=true"></a></p>
## Encodings
All encodings from the Encoding specification are supported:
utf-8 ibm866 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-8-i iso-8859-10 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 koi8-r koi8-u macintosh windows-874 windows-1250 windows-1251 windows-1252 windows-1253 windows-1254 windows-1255 windows-1256 windows-1257 windows-1258 x-mac-cyrillic gb18030 hz-gb-2312 big5 euc-jp iso-2022-jp shift_jis euc-kr replacement utf-16be utf-16le x-user-defined
(Some encodings may be supported under other names, e.g. ascii, iso-8859-1, etc. See [Encoding](https://encoding.spec.whatwg.org/) for additional labels for each encoding.)
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/6.svg?sanitize=true"></a></p>
## Copyright
Original Work By [Joshua Bell](https://github.com/inexorabletash/text-encoding) under dual Unlicense/Apache-2.0 license.
> The encoding indexes, algorithms, and many comments in the code derive from the Encoding Standard https://encoding.spec.whatwg.org/
---
<table>
<tr>
<th>
<a href="https://artd.eco">
<img src="https://raw.githubusercontent.com/wrote/wrote/master/images/artdeco.png" alt="Art Deco">
</a>
</th>
<th>© <a href="https://artd.eco">Art Deco</a> for <a href="https://idio.cc">Idio</a> 2019</th>
<th>
<a href="https://idio.cc">
<img src="https://avatars3.githubusercontent.com/u/40834161?s=100" width="100" alt="Idio">
</a>
</th>
<th>
<a href="https://www.technation.sucks" title="Tech Nation Visa">
<img src="https://raw.githubusercontent.com/artdecoweb/www.technation.sucks/master/anim.gif"
alt="Tech Nation Visa">
</a>
</th>
<th><a href="https://www.technation.sucks">Tech Nation Visa Sucks</a></th>
</tr>
</table>
<p align="center"><a href="#table-of-contents"><img src="/.documentary/section-breaks/-1.svg?sanitize=true"></a></p>
File diff suppressed because one or more lines are too long
+460
View File
@@ -0,0 +1,460 @@
/**
* Encodings table: https://encoding.spec.whatwg.org/encodings.json
*/
const encodings = [
{
encodings: [
{
labels: [
"unicode-1-1-utf-8",
"utf-8",
"utf8",
],
name: "UTF-8",
},
],
heading: "The Encoding",
},
{
encodings: [
{
labels: [
"866",
"cp866",
"csibm866",
"ibm866",
],
name: "IBM866",
},
{
labels: [
"csisolatin2",
"iso-8859-2",
"iso-ir-101",
"iso8859-2",
"iso88592",
"iso_8859-2",
"iso_8859-2:1987",
"l2",
"latin2",
],
name: "ISO-8859-2",
},
{
labels: [
"csisolatin3",
"iso-8859-3",
"iso-ir-109",
"iso8859-3",
"iso88593",
"iso_8859-3",
"iso_8859-3:1988",
"l3",
"latin3",
],
name: "ISO-8859-3",
},
{
labels: [
"csisolatin4",
"iso-8859-4",
"iso-ir-110",
"iso8859-4",
"iso88594",
"iso_8859-4",
"iso_8859-4:1988",
"l4",
"latin4",
],
name: "ISO-8859-4",
},
{
labels: [
"csisolatincyrillic",
"cyrillic",
"iso-8859-5",
"iso-ir-144",
"iso8859-5",
"iso88595",
"iso_8859-5",
"iso_8859-5:1988",
],
name: "ISO-8859-5",
},
{
labels: [
"arabic",
"asmo-708",
"csiso88596e",
"csiso88596i",
"csisolatinarabic",
"ecma-114",
"iso-8859-6",
"iso-8859-6-e",
"iso-8859-6-i",
"iso-ir-127",
"iso8859-6",
"iso88596",
"iso_8859-6",
"iso_8859-6:1987",
],
name: "ISO-8859-6",
},
{
labels: [
"csisolatingreek",
"ecma-118",
"elot_928",
"greek",
"greek8",
"iso-8859-7",
"iso-ir-126",
"iso8859-7",
"iso88597",
"iso_8859-7",
"iso_8859-7:1987",
"sun_eu_greek",
],
name: "ISO-8859-7",
},
{
labels: [
"csiso88598e",
"csisolatinhebrew",
"hebrew",
"iso-8859-8",
"iso-8859-8-e",
"iso-ir-138",
"iso8859-8",
"iso88598",
"iso_8859-8",
"iso_8859-8:1988",
"visual",
],
name: "ISO-8859-8",
},
{
labels: [
"csiso88598i",
"iso-8859-8-i",
"logical",
],
name: "ISO-8859-8-I",
},
{
labels: [
"csisolatin6",
"iso-8859-10",
"iso-ir-157",
"iso8859-10",
"iso885910",
"l6",
"latin6",
],
name: "ISO-8859-10",
},
{
labels: [
"iso-8859-13",
"iso8859-13",
"iso885913",
],
name: "ISO-8859-13",
},
{
labels: [
"iso-8859-14",
"iso8859-14",
"iso885914",
],
name: "ISO-8859-14",
},
{
labels: [
"csisolatin9",
"iso-8859-15",
"iso8859-15",
"iso885915",
"iso_8859-15",
"l9",
],
name: "ISO-8859-15",
},
{
labels: [
"iso-8859-16",
],
name: "ISO-8859-16",
},
{
labels: [
"cskoi8r",
"koi",
"koi8",
"koi8-r",
"koi8_r",
],
name: "KOI8-R",
},
{
labels: [
"koi8-ru",
"koi8-u",
],
name: "KOI8-U",
},
{
labels: [
"csmacintosh",
"mac",
"macintosh",
"x-mac-roman",
],
name: "macintosh",
},
{
labels: [
"dos-874",
"iso-8859-11",
"iso8859-11",
"iso885911",
"tis-620",
"windows-874",
],
name: "windows-874",
},
{
labels: [
"cp1250",
"windows-1250",
"x-cp1250",
],
name: "windows-1250",
},
{
labels: [
"cp1251",
"windows-1251",
"x-cp1251",
],
name: "windows-1251",
},
{
labels: [
"ansi_x3.4-1968",
"ascii",
"cp1252",
"cp819",
"csisolatin1",
"ibm819",
"iso-8859-1",
"iso-ir-100",
"iso8859-1",
"iso88591",
"iso_8859-1",
"iso_8859-1:1987",
"l1",
"latin1",
"us-ascii",
"windows-1252",
"x-cp1252",
],
name: "windows-1252",
},
{
labels: [
"cp1253",
"windows-1253",
"x-cp1253",
],
name: "windows-1253",
},
{
labels: [
"cp1254",
"csisolatin5",
"iso-8859-9",
"iso-ir-148",
"iso8859-9",
"iso88599",
"iso_8859-9",
"iso_8859-9:1989",
"l5",
"latin5",
"windows-1254",
"x-cp1254",
],
name: "windows-1254",
},
{
labels: [
"cp1255",
"windows-1255",
"x-cp1255",
],
name: "windows-1255",
},
{
labels: [
"cp1256",
"windows-1256",
"x-cp1256",
],
name: "windows-1256",
},
{
labels: [
"cp1257",
"windows-1257",
"x-cp1257",
],
name: "windows-1257",
},
{
labels: [
"cp1258",
"windows-1258",
"x-cp1258",
],
name: "windows-1258",
},
{
labels: [
"x-mac-cyrillic",
"x-mac-ukrainian",
],
name: "x-mac-cyrillic",
},
],
heading: "Legacy single-byte encodings",
},
{
encodings: [
{
labels: [
"chinese",
"csgb2312",
"csiso58gb231280",
"gb2312",
"gb_2312",
"gb_2312-80",
"gbk",
"iso-ir-58",
"x-gbk",
],
name: "GBK",
},
{
labels: [
"gb18030",
],
name: "gb18030",
},
],
heading: "Legacy multi-byte Chinese (simplified) encodings",
},
{
encodings: [
{
labels: [
"big5",
"big5-hkscs",
"cn-big5",
"csbig5",
"x-x-big5",
],
name: "Big5",
},
],
heading: "Legacy multi-byte Chinese (traditional) encodings",
},
{
encodings: [
{
labels: [
"cseucpkdfmtjapanese",
"euc-jp",
"x-euc-jp",
],
name: "EUC-JP",
},
{
labels: [
"csiso2022jp",
"iso-2022-jp",
],
name: "ISO-2022-JP",
},
{
labels: [
"csshiftjis",
"ms932",
"ms_kanji",
"shift-jis",
"shift_jis",
"sjis",
"windows-31j",
"x-sjis",
],
name: "Shift_JIS",
},
],
heading: "Legacy multi-byte Japanese encodings",
},
{
encodings: [
{
labels: [
"cseuckr",
"csksc56011987",
"euc-kr",
"iso-ir-149",
"korean",
"ks_c_5601-1987",
"ks_c_5601-1989",
"ksc5601",
"ksc_5601",
"windows-949",
],
name: "EUC-KR",
},
],
heading: "Legacy multi-byte Korean encodings",
},
{
encodings: [
{
labels: [
"csiso2022kr",
"hz-gb-2312",
"iso-2022-cn",
"iso-2022-cn-ext",
"iso-2022-kr",
],
name: "replacement",
},
{
labels: [
"utf-16be",
],
name: "UTF-16BE",
},
{
labels: [
"utf-16",
"utf-16le",
],
name: "UTF-16LE",
},
{
labels: [
"x-user-defined",
],
name: "x-user-defined",
},
],
heading: "Legacy miscellaneous encodings",
},
]
module.exports=encodings
+157
View File
@@ -0,0 +1,157 @@
const { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } = require('../utils');
const index = require('../indexes'); const { indexBig5PointerFor, indexCodePointFor } = index;
//
// 12. Legacy multi-byte Chinese (traditional) encodings
//
// 12.1 Big5
// 12.1.1 Big5 decoder
/**
* @implements {Decoder}
*/
class Big5Decoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// Big5's decoder has an associated Big5 lead (initially 0x00).
this.Big5_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and Big5 lead is not 0x00, set
// Big5 lead to 0x00 and return error.
if (bite === end_of_stream && this.Big5_lead !== 0x00) {
this.Big5_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and Big5 lead is 0x00, return
// finished.
if (bite === end_of_stream && this.Big5_lead === 0x00)
return finished
// 3. If Big5 lead is not 0x00, let lead be Big5 lead, let
// pointer be null, set Big5 lead to 0x00, and then run these
// substeps:
if (this.Big5_lead !== 0x00) {
const lead = this.Big5_lead
let pointer = null
this.Big5_lead = 0x00
// 1. Let offset be 0x40 if byte is less than 0x7F and 0x62
// otherwise.
const offset = bite < 0x7F ? 0x40 : 0x62
// 2. If byte is in the range 0x40 to 0x7E, inclusive, or 0xA1
// to 0xFE, inclusive, set pointer to (lead 0x81) × 157 +
// (byte offset).
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0xA1, 0xFE))
pointer = (lead - 0x81) * 157 + (bite - offset)
// 3. If there is a row in the table below whose first column
// is pointer, return the two code points listed in its second
// column
// Pointer | Code points
// --------+--------------
// 1133 | U+00CA U+0304
// 1135 | U+00CA U+030C
// 1164 | U+00EA U+0304
// 1166 | U+00EA U+030C
switch (pointer) {
case 1133: return [0x00CA, 0x0304]
case 1135: return [0x00CA, 0x030C]
case 1164: return [0x00EA, 0x0304]
case 1166: return [0x00EA, 0x030C]
}
// 4. Let code point be null if pointer is null and the index
// code point for pointer in index Big5 otherwise.
const code_point = (pointer === null) ? null :
indexCodePointFor(pointer, index('big5'))
// 5. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 6. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 7. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 5. If byte is in the range 0x81 to 0xFE, inclusive, set Big5
// lead to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.Big5_lead = bite
return null
}
// 6. Return error.
return decoderError(this.fatal)
}
}
// 12.1.2 Big5 encoder
/**
* @implements {Encoder}
*/
class Big5Encoder {
constructor() {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
this.handler = function(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index Big5 pointer for code point.
const pointer = indexBig5PointerFor(code_point)
// 4. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 5. Let lead be floor(pointer / 157) + 0x81.
const lead = floor(pointer / 157) + 0x81
// 6. If lead is less than 0xA1, return error with code point.
if (lead < 0xA1)
return encoderError(code_point)
// 7. Let trail be pointer % 157.
const trail = pointer % 157
// 8. Let offset be 0x40 if trail is less than 0x3F and 0x62
// otherwise.
const offset = trail < 0x3F ? 0x40 : 0x62
// Return two bytes whose values are lead and trail + offset.
return [lead, trail + offset]
}
}
}
module.exports.Big5Decoder = Big5Decoder
module.exports.Big5Encoder = Big5Encoder
+169
View File
@@ -0,0 +1,169 @@
const { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } = require('../utils');
const index = require('../indexes'); const { indexCodePointFor, indexPointerFor } = index;
//
// 13. Legacy multi-byte Japanese encodings
//
// 13.1 euc-jp
// 13.1.1 euc-jp decoder
/**
* @implements {Decoder}
*/
class EUCJPDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// euc-jp's decoder has an associated euc-jp jis0212 flag
// (initially unset) and euc-jp lead (initially 0x00).
this.eucjp_jis0212_flag = false
this.eucjp_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and euc-jp lead is not 0x00, set
// euc-jp lead to 0x00, and return error.
if (bite === end_of_stream && this.eucjp_lead !== 0x00) {
this.eucjp_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and euc-jp lead is 0x00, return
// finished.
if (bite === end_of_stream && this.eucjp_lead === 0x00)
return finished
// 3. If euc-jp lead is 0x8E and byte is in the range 0xA1 to
// 0xDF, inclusive, set euc-jp lead to 0x00 and return a code
// point whose value is 0xFF61 0xA1 + byte.
if (this.eucjp_lead === 0x8E && inRange(bite, 0xA1, 0xDF)) {
this.eucjp_lead = 0x00
return 0xFF61 - 0xA1 + bite
}
// 4. If euc-jp lead is 0x8F and byte is in the range 0xA1 to
// 0xFE, inclusive, set the euc-jp jis0212 flag, set euc-jp lead
// to byte, and return continue.
if (this.eucjp_lead === 0x8F && inRange(bite, 0xA1, 0xFE)) {
this.eucjp_jis0212_flag = true
this.eucjp_lead = bite
return null
}
// 5. If euc-jp lead is not 0x00, let lead be euc-jp lead, set
// euc-jp lead to 0x00, and run these substeps:
if (this.eucjp_lead !== 0x00) {
const lead = this.eucjp_lead
this.eucjp_lead = 0x00
// 1. Let code point be null.
let code_point = null
// 2. If lead and byte are both in the range 0xA1 to 0xFE,
// inclusive, set code point to the index code point for (lead
// 0xA1) × 94 + byte 0xA1 in index jis0208 if the euc-jp
// jis0212 flag is unset and in index jis0212 otherwise.
if (inRange(lead, 0xA1, 0xFE) && inRange(bite, 0xA1, 0xFE)) {
code_point = indexCodePointFor(
(lead - 0xA1) * 94 + (bite - 0xA1),
index(!this.eucjp_jis0212_flag ? 'jis0208' : 'jis0212'))
}
// 3. Unset the euc-jp jis0212 flag.
this.eucjp_jis0212_flag = false
// 4. If byte is not in the range 0xA1 to 0xFE, inclusive,
// prepend byte to stream.
if (!inRange(bite, 0xA1, 0xFE))
stream.prepend(bite)
// 5. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 6. Return a code point whose value is code point.
return code_point
}
// 6. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 7. If byte is 0x8E, 0x8F, or in the range 0xA1 to 0xFE,
// inclusive, set euc-jp lead to byte and return continue.
if (bite === 0x8E || bite === 0x8F || inRange(bite, 0xA1, 0xFE)) {
this.eucjp_lead = bite
return null
}
// 8. Return error.
return decoderError(this.fatal)
}
}
// 13.1.2 euc-jp encoder
/**
* @implements {Encoder}
*/
class EUCJPEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 4. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
// 5. If code point is in the range U+FF61 to U+FF9F, inclusive,
// return two bytes whose values are 0x8E and code point
// 0xFF61 + 0xA1.
if (inRange(code_point, 0xFF61, 0xFF9F))
return [0x8E, code_point - 0xFF61 + 0xA1]
// 6. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 7. Let pointer be the index pointer for code point in index
// jis0208.
const pointer = indexPointerFor(code_point, index('jis0208'))
// 8. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 9. Let lead be floor(pointer / 94) + 0xA1.
const lead = floor(pointer / 94) + 0xA1
// 10. Let trail be pointer % 94 + 0xA1.
const trail = pointer % 94 + 0xA1
// 11. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
module.exports.EUCJPDecoder = EUCJPDecoder
module.exports.EUCJPEncoder = EUCJPEncoder
+127
View File
@@ -0,0 +1,127 @@
const { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } = require('../utils');
const index = require('../indexes'); const { indexCodePointFor, indexPointerFor } = index;
//
// 14. Legacy multi-byte Korean encodings
//
// 14.1 euc-kr
// 14.1.1 euc-kr decoder
/**
* @implements {Decoder}
*/
class EUCKRDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// euc-kr's decoder has an associated euc-kr lead (initially 0x00).
this.euckr_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and euc-kr lead is not 0x00, set
// euc-kr lead to 0x00 and return error.
if (bite === end_of_stream && this.euckr_lead !== 0) {
this.euckr_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and euc-kr lead is 0x00, return
// finished.
if (bite === end_of_stream && this.euckr_lead === 0)
return finished
// 3. If euc-kr lead is not 0x00, let lead be euc-kr lead, let
// pointer be null, set euc-kr lead to 0x00, and then run these
// substeps:
if (this.euckr_lead !== 0x00) {
const lead = this.euckr_lead
let pointer = null
this.euckr_lead = 0x00
// 1. If byte is in the range 0x41 to 0xFE, inclusive, set
// pointer to (lead 0x81) × 190 + (byte 0x41).
if (inRange(bite, 0x41, 0xFE))
pointer = (lead - 0x81) * 190 + (bite - 0x41)
// 2. Let code point be null, if pointer is null, and the
// index code point for pointer in index euc-kr otherwise.
const code_point = (pointer === null)
? null : indexCodePointFor(pointer, index('euc-kr'))
// 3. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (pointer === null && isASCIIByte(bite))
stream.prepend(bite)
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 5. If byte is in the range 0x81 to 0xFE, inclusive, set
// euc-kr lead to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.euckr_lead = bite
return null
}
// 6. Return error.
return decoderError(this.fatal)
}
}
// 14.1.2 euc-kr encoder
/**
* @implements {Encoder}
*/
class EUCKREncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index pointer for code point in index
// euc-kr.
const pointer = indexPointerFor(code_point, index('euc-kr'))
// 4. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 5. Let lead be floor(pointer / 190) + 0x81.
const lead = floor(pointer / 190) + 0x81
// 6. Let trail be pointer % 190 + 0x41.
const trail = (pointer % 190) + 0x41
// 7. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
module.exports.EUCKRDecoder = EUCKRDecoder
module.exports.EUCKREncoder = EUCKREncoder
+255
View File
@@ -0,0 +1,255 @@
const { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } = require('../utils');
const index = require('../indexes'); const {
indexGB18030RangesCodePointFor, indexGB18030RangesPointerFor,
indexCodePointFor, indexPointerFor } = index;
// 11.2 gb18030
// 11.2.1 gb18030 decoder
/**
* @constructor
* @implements {Decoder}
* @param {{fatal: boolean}} options
*/
class GB18030Decoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// gb18030's decoder has an associated gb18030 first, gb18030
// second, and gb18030 third (all initially 0x00).
this.gb18030_first = 0x00
this.gb18030_second = 0x00,
this.gb18030_third = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
* @return The next code point(s) decoded, or null if not enough data exists in the input stream to decode a complete code point.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and gb18030 first, gb18030
// second, and gb18030 third are 0x00, return finished.
if (bite === end_of_stream && this.gb18030_first === 0x00 &&
this.gb18030_second === 0x00 && this.gb18030_third === 0x00) {
return finished
}
// 2. If byte is end-of-stream, and gb18030 first, gb18030
// second, or gb18030 third is not 0x00, set gb18030 first,
// gb18030 second, and gb18030 third to 0x00, and return error.
if (bite === end_of_stream &&
(this.gb18030_first !== 0x00 || this.gb18030_second !== 0x00 ||
this.gb18030_third !== 0x00)) {
this.gb18030_first = 0x00
this.gb18030_second = 0x00
this.gb18030_third = 0x00
decoderError(this.fatal)
}
var code_point
// 3. If gb18030 third is not 0x00, run these substeps:
if (this.gb18030_third !== 0x00) {
// 1. Let code point be null.
code_point = null
// 2. If byte is in the range 0x30 to 0x39, inclusive, set
// code point to the index gb18030 ranges code point for
// (((gb18030 first 0x81) × 10 + gb18030 second 0x30) ×
// 126 + gb18030 third 0x81) × 10 + byte 0x30.
if (inRange(bite, 0x30, 0x39)) {
code_point = indexGB18030RangesCodePointFor(
(((this.gb18030_first - 0x81) * 10 + this.gb18030_second - 0x30) * 126 +
this.gb18030_third - 0x81) * 10 + bite - 0x30)
}
// 3. Let buffer be a byte sequence consisting of gb18030
// second, gb18030 third, and byte, in order.
var buffer = [this.gb18030_second, this.gb18030_third, bite]
// 4. Set gb18030 first, gb18030 second, and gb18030 third to
// 0x00.
this.gb18030_first = 0x00
this.gb18030_second = 0x00
this.gb18030_third = 0x00
// 5. If code point is null, prepend buffer to stream and
// return error.
if (code_point === null) {
stream.prepend(buffer)
return decoderError(this.fatal)
}
// 6. Return a code point whose value is code point.
return code_point
}
// 4. If gb18030 second is not 0x00, run these substeps:
if (this.gb18030_second !== 0x00) {
// 1. If byte is in the range 0x81 to 0xFE, inclusive, set
// gb18030 third to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.gb18030_third = bite
return null
}
// 2. Prepend gb18030 second followed by byte to stream, set
// gb18030 first and gb18030 second to 0x00, and return error.
stream.prepend([this.gb18030_second, bite])
this.gb18030_first = 0x00
this.gb18030_second = 0x00
return decoderError(this.fatal)
}
// 5. If gb18030 first is not 0x00, run these substeps:
if (this.gb18030_first !== 0x00) {
// 1. If byte is in the range 0x30 to 0x39, inclusive, set
// gb18030 second to byte and return continue.
if (inRange(bite, 0x30, 0x39)) {
this.gb18030_second = bite
return null
}
// 2. Let lead be gb18030 first, let pointer be null, and set
// gb18030 first to 0x00.
var lead = this.gb18030_first
var pointer = null
this.gb18030_first = 0x00
// 3. Let offset be 0x40 if byte is less than 0x7F and 0x41
// otherwise.
var offset = bite < 0x7F ? 0x40 : 0x41
// 4. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80
// to 0xFE, inclusive, set pointer to (lead 0x81) × 190 +
// (byte offset).
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0x80, 0xFE))
pointer = (lead - 0x81) * 190 + (bite - offset)
// 5. Let code point be null if pointer is null and the index
// code point for pointer in index gb18030 otherwise.
code_point = pointer === null ? null :
indexCodePointFor(pointer, index('gb18030'))
// 6. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 7. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 8. Return a code point whose value is code point.
return code_point
}
// 6. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 7. If byte is 0x80, return code point U+20AC.
if (bite === 0x80)
return 0x20AC
// 8. If byte is in the range 0x81 to 0xFE, inclusive, set
// gb18030 first to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.gb18030_first = bite
return null
}
// 9. Return error.
return decoderError(this.fatal)
}
}
// 11.2.2 gb18030 encoder
/**
* @implements {Encoder}
*/
class GB18030Encoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is U+E5E5, return error with code point.
if (code_point === 0xE5E5)
return encoderError(code_point)
// 4. If the gbk flag is set and code point is U+20AC, return
// byte 0x80.
if (this.gbk_flag && code_point === 0x20AC)
return 0x80
// 5. Let pointer be the index pointer for code point in index
// gb18030.
var pointer = indexPointerFor(code_point, index('gb18030'))
// 6. If pointer is not null, run these substeps:
if (pointer !== null) {
// 1. Let lead be floor(pointer / 190) + 0x81.
var lead = floor(pointer / 190) + 0x81
// 2. Let trail be pointer % 190.
var trail = pointer % 190
// 3. Let offset be 0x40 if trail is less than 0x3F and 0x41 otherwise.
var offset = trail < 0x3F ? 0x40 : 0x41
// 4. Return two bytes whose values are lead and trail + offset.
return [lead, trail + offset]
}
// 7. If gbk flag is set, return error with code point.
if (this.gbk_flag)
return encoderError(code_point)
// 8. Set pointer to the index gb18030 ranges pointer for code
// point.
pointer = indexGB18030RangesPointerFor(code_point)
// 9. Let byte1 be floor(pointer / 10 / 126 / 10).
var byte1 = floor(pointer / 10 / 126 / 10)
// 10. Set pointer to pointer byte1 × 10 × 126 × 10.
pointer = pointer - byte1 * 10 * 126 * 10
// 11. Let byte2 be floor(pointer / 10 / 126).
var byte2 = floor(pointer / 10 / 126)
// 12. Set pointer to pointer byte2 × 10 × 126.
pointer = pointer - byte2 * 10 * 126
// 13. Let byte3 be floor(pointer / 10).
var byte3 = floor(pointer / 10)
// 14. Let byte4 be pointer byte3 × 10.
var byte4 = pointer - byte3 * 10
// 15. Return four bytes whose values are byte1 + 0x81, byte2 +
// 0x30, byte3 + 0x81, byte4 + 0x30.
return [byte1 + 0x81,
byte2 + 0x30,
byte3 + 0x81,
byte4 + 0x30]
}
constructor(options = {}, gbk_flag = false) {
// gb18030's decoder has an associated gbk flag (initially unset).
this.gbk_flag = gbk_flag
}
}
module.exports.GB18030Decoder = GB18030Decoder
module.exports.GB18030Encoder = GB18030Encoder
+444
View File
@@ -0,0 +1,444 @@
const { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, floor } = require('../utils');
const index = require('../indexes'); const { indexCodePointFor, indexPointerFor } = index;
// 13.2 iso-2022-jp
// 13.2.1 iso-2022-jp decoder
/**
* @implements {Decoder}
*/
class ISO2022JPDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
/** @enum */
this.states = {
ASCII: 0,
Roman: 1,
Katakana: 2,
LeadByte: 3,
TrailByte: 4,
EscapeStart: 5,
Escape: 6,
}
// iso-2022-jp's decoder has an associated iso-2022-jp decoder
// state (initially ASCII), iso-2022-jp decoder output state
// (initially ASCII), iso-2022-jp lead (initially 0x00), and
// iso-2022-jp output flag (initially unset).
this.iso2022jp_decoder_state = this.states.ASCII
this.iso2022jp_decoder_output_state = this.states.ASCII,
this.iso2022jp_lead = 0x00
this.iso2022jp_output_flag = false
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// switching on iso-2022-jp decoder state:
switch (this.iso2022jp_decoder_state) {
default:
case this.states.ASCII:
// ASCII
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x00 to 0x7F, excluding 0x0E, 0x0F, and 0x1B
if (inRange(bite, 0x00, 0x7F) && bite !== 0x0E
&& bite !== 0x0F && bite !== 0x1B) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is byte.
this.iso2022jp_output_flag = false
return bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.Roman:
// Roman
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x5C
if (bite === 0x5C) {
// Unset the iso-2022-jp output flag and return code point
// U+00A5.
this.iso2022jp_output_flag = false
return 0x00A5
}
// 0x7E
if (bite === 0x7E) {
// Unset the iso-2022-jp output flag and return code point
// U+203E.
this.iso2022jp_output_flag = false
return 0x203E
}
// 0x00 to 0x7F, excluding 0x0E, 0x0F, 0x1B, 0x5C, and 0x7E
if (inRange(bite, 0x00, 0x7F) && bite !== 0x0E && bite !== 0x0F
&& bite !== 0x1B && bite !== 0x5C && bite !== 0x7E) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is byte.
this.iso2022jp_output_flag = false
return bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.Katakana:
// Katakana
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x21 to 0x5F
if (inRange(bite, 0x21, 0x5F)) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is 0xFF61 0x21 + byte.
this.iso2022jp_output_flag = false
return 0xFF61 - 0x21 + bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.LeadByte:
// Lead byte
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x21 to 0x7E
if (inRange(bite, 0x21, 0x7E)) {
// Unset the iso-2022-jp output flag, set iso-2022-jp lead
// to byte, iso-2022-jp decoder state to trail byte, and
// return continue.
this.iso2022jp_output_flag = false
this.iso2022jp_lead = bite
this.iso2022jp_decoder_state = this.states.TrailByte
return null
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.TrailByte:
// Trail byte
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return decoderError(this.fatal)
}
// 0x21 to 0x7E
if (inRange(bite, 0x21, 0x7E)) {
// 1. Set the iso-2022-jp decoder state to lead byte.
this.iso2022jp_decoder_state = this.states.LeadByte
// 2. Let pointer be (iso-2022-jp lead 0x21) × 94 + byte 0x21.
const pointer = (this.iso2022jp_lead - 0x21) * 94 + bite - 0x21
// 3. Let code point be the index code point for pointer in
// index jis0208.
const code_point = indexCodePointFor(pointer, index('jis0208'))
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
// end-of-stream
if (bite === end_of_stream) {
// Set the iso-2022-jp decoder state to lead byte, prepend
// byte to stream, and return error.
this.iso2022jp_decoder_state = this.states.LeadByte
stream.prepend(bite)
return decoderError(this.fatal)
}
// Otherwise
// Set iso-2022-jp decoder state to lead byte and return
// error.
this.iso2022jp_decoder_state = this.states.LeadByte
return decoderError(this.fatal)
case this.states.EscapeStart:
// Escape start
// 1. If byte is either 0x24 or 0x28, set iso-2022-jp lead to
// byte, iso-2022-jp decoder state to escape, and return
// continue.
if (bite === 0x24 || bite === 0x28) {
this.iso2022jp_lead = bite
this.iso2022jp_decoder_state = this.states.Escape
return null
}
// 2. Prepend byte to stream.
stream.prepend(bite)
// 3. Unset the iso-2022-jp output flag, set iso-2022-jp
// decoder state to iso-2022-jp decoder output state, and
// return error.
this.iso2022jp_output_flag = false
this.iso2022jp_decoder_state = this.iso2022jp_decoder_output_state
return decoderError(this.fatal)
case this.states.Escape: {
// Escape
// 1. Let lead be iso-2022-jp lead and set iso-2022-jp lead to
// 0x00.
const lead = this.iso2022jp_lead
this.iso2022jp_lead = 0x00
// 2. Let state be null.
let state = null
// 3. If lead is 0x28 and byte is 0x42, set state to ASCII.
if (lead === 0x28 && bite === 0x42)
state = this.states.ASCII
// 4. If lead is 0x28 and byte is 0x4A, set state to Roman.
if (lead === 0x28 && bite === 0x4A)
state = this.states.Roman
// 5. If lead is 0x28 and byte is 0x49, set state to Katakana.
if (lead === 0x28 && bite === 0x49)
state = this.states.Katakana
// 6. If lead is 0x24 and byte is either 0x40 or 0x42, set
// state to lead byte.
if (lead === 0x24 && (bite === 0x40 || bite === 0x42))
state = this.states.LeadByte
// 7. If state is non-null, run these substeps:
if (state !== null) {
// 1. Set iso-2022-jp decoder state and iso-2022-jp decoder
// output state to this.states.
this.iso2022jp_decoder_state = this.iso2022jp_decoder_state = state
// 2. Let output flag be the iso-2022-jp output flag.
const output_flag = this.iso2022jp_output_flag
// 3. Set the iso-2022-jp output flag.
this.iso2022jp_output_flag = true
// 4. Return continue, if output flag is unset, and error
// otherwise.
return !output_flag ? null : decoderError(this.fatal)
}
// 8. Prepend lead and byte to stream.
stream.prepend([lead, bite])
// 9. Unset the iso-2022-jp output flag, set iso-2022-jp
// decoder state to iso-2022-jp decoder output state and
// return error.
this.iso2022jp_output_flag = false
this.iso2022jp_decoder_state = this.iso2022jp_decoder_output_state
return decoderError(this.fatal)
}
}
}
}
// 13.2.2 iso-2022-jp encoder
/**
* @implements {Encoder}
*/
class ISO2022JPEncoder {
constructor() {
// iso-2022-jp's encoder has an associated iso-2022-jp encoder
// state which is one of ASCII, Roman, and jis0208 (initially
// ASCII).
/** @enum */
this.states = {
ASCII: 0,
Roman: 1,
jis0208: 2,
}
this.iso2022jp_state = this.states.ASCII
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream and iso-2022-jp encoder
// state is not ASCII, prepend code point to stream, set
// iso-2022-jp encoder state to ASCII, and return three bytes
// 0x1B 0x28 0x42.
if (code_point === end_of_stream &&
this.iso2022jp_state !== this.states.ASCII) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.ASCII
return [0x1B, 0x28, 0x42]
}
// 2. If code point is end-of-stream and iso-2022-jp encoder
// state is ASCII, return finished.
if (code_point === end_of_stream && this.iso2022jp_state === this.states.ASCII)
return finished
// 3. If ISO-2022-JP encoder state is ASCII or Roman, and code
// point is U+000E, U+000F, or U+001B, return error with U+FFFD.
if ((this.iso2022jp_state === this.states.ASCII ||
this.iso2022jp_state === this.states.Roman) &&
(code_point === 0x000E || code_point === 0x000F ||
code_point === 0x001B)) {
return encoderError(0xFFFD)
}
// 4. If iso-2022-jp encoder state is ASCII and code point is an
// ASCII code point, return a byte whose value is code point.
if (this.iso2022jp_state === this.states.ASCII &&
isASCIICodePoint(code_point))
return code_point
// 5. If iso-2022-jp encoder state is Roman and code point is an
// ASCII code point, excluding U+005C and U+007E, or is U+00A5
// or U+203E, run these substeps:
if (this.iso2022jp_state === this.states.Roman &&
((isASCIICodePoint(code_point) &&
code_point !== 0x005C && code_point !== 0x007E) ||
(code_point == 0x00A5 || code_point == 0x203E))) {
// 1. If code point is an ASCII code point, return a byte
// whose value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 2. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 3. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
}
// 6. If code point is an ASCII code point, and iso-2022-jp
// encoder state is not ASCII, prepend code point to stream, set
// iso-2022-jp encoder state to ASCII, and return three bytes
// 0x1B 0x28 0x42.
if (isASCIICodePoint(code_point) &&
this.iso2022jp_state !== this.states.ASCII) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.ASCII
return [0x1B, 0x28, 0x42]
}
// 7. If code point is either U+00A5 or U+203E, and iso-2022-jp
// encoder state is not Roman, prepend code point to stream, set
// iso-2022-jp encoder state to Roman, and return three bytes
// 0x1B 0x28 0x4A.
if ((code_point === 0x00A5 || code_point === 0x203E) &&
this.iso2022jp_state !== this.states.Roman) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.Roman
return [0x1B, 0x28, 0x4A]
}
// 8. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 9. Let pointer be the index pointer for code point in index
// jis0208.
const pointer = indexPointerFor(code_point, index('jis0208'))
// 10. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 11. If iso-2022-jp encoder state is not jis0208, prepend code
// point to stream, set iso-2022-jp encoder state to jis0208,
// and return three bytes 0x1B 0x24 0x42.
if (this.iso2022jp_state !== this.states.jis0208) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.jis0208
return [0x1B, 0x24, 0x42]
}
// 12. Let lead be floor(pointer / 94) + 0x21.
const lead = floor(pointer / 94) + 0x21
// 13. Let trail be pointer % 94 + 0x21.
const trail = pointer % 94 + 0x21
// 14. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
module.exports.ISO2022JPDecoder = ISO2022JPDecoder
module.exports.ISO2022JPEncoder = ISO2022JPEncoder
+173
View File
@@ -0,0 +1,173 @@
const { inRange, decoderError, encoderError, floor, isASCIICodePoint, isASCIIByte,
end_of_stream, finished } = require('../utils');
const index = require('../indexes'); const { indexCodePointFor, indexShiftJISPointerFor } = index;
// 13.3 Shift_JIS
// 13.3.1 Shift_JIS decoder
/**
* @constructor
* @implements {Decoder}
* @param {{fatal: boolean}} options
*/
class ShiftJISDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// Shift_JIS's decoder has an associated Shift_JIS lead (initially
// 0x00).
this.Shift_JIS_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and Shift_JIS lead is not 0x00,
// set Shift_JIS lead to 0x00 and return error.
if (bite === end_of_stream && this.Shift_JIS_lead !== 0x00) {
this.Shift_JIS_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and Shift_JIS lead is 0x00,
// return finished.
if (bite === end_of_stream && this.Shift_JIS_lead === 0x00)
return finished
// 3. If Shift_JIS lead is not 0x00, let lead be Shift_JIS lead,
// let pointer be null, set Shift_JIS lead to 0x00, and then run
// these substeps:
if (this.Shift_JIS_lead !== 0x00) {
var lead = this.Shift_JIS_lead
var pointer = null
this.Shift_JIS_lead = 0x00
// 1. Let offset be 0x40, if byte is less than 0x7F, and 0x41
// otherwise.
var offset = (bite < 0x7F) ? 0x40 : 0x41
// 2. Let lead offset be 0x81, if lead is less than 0xA0, and
// 0xC1 otherwise.
var lead_offset = (lead < 0xA0) ? 0x81 : 0xC1
// 3. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80
// to 0xFC, inclusive, set pointer to (lead lead offset) ×
// 188 + byte offset.
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0x80, 0xFC))
pointer = (lead - lead_offset) * 188 + bite - offset
// 4. If pointer is in the range 8836 to 10715, inclusive,
// return a code point whose value is 0xE000 8836 + pointer.
if (inRange(pointer, 8836, 10715))
return 0xE000 - 8836 + pointer
// 5. Let code point be null, if pointer is null, and the
// index code point for pointer in index jis0208 otherwise.
var code_point = (pointer === null) ? null :
indexCodePointFor(pointer, index('jis0208'))
// 6. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 7. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 8. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte or 0x80, return a code point
// whose value is byte.
if (isASCIIByte(bite) || bite === 0x80)
return bite
// 5. If byte is in the range 0xA1 to 0xDF, inclusive, return a
// code point whose value is 0xFF61 0xA1 + byte.
if (inRange(bite, 0xA1, 0xDF))
return 0xFF61 - 0xA1 + bite
// 6. If byte is in the range 0x81 to 0x9F, inclusive, or 0xE0
// to 0xFC, inclusive, set Shift_JIS lead to byte and return
// continue.
if (inRange(bite, 0x81, 0x9F) || inRange(bite, 0xE0, 0xFC)) {
this.Shift_JIS_lead = bite
return null
}
// 7. Return error.
return decoderError(this.fatal)
}
}
// 13.3.2 Shift_JIS encoder
/**
* @constructor
* @implements {Encoder}
* @param {{fatal: boolean}} options
*/
class ShiftJISEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point or U+0080, return a
// byte whose value is code point.
if (isASCIICodePoint(code_point) || code_point === 0x0080)
return code_point
// 3. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 4. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
// 5. If code point is in the range U+FF61 to U+FF9F, inclusive,
// return a byte whose value is code point 0xFF61 + 0xA1.
if (inRange(code_point, 0xFF61, 0xFF9F))
return code_point - 0xFF61 + 0xA1
// 6. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 7. Let pointer be the index Shift_JIS pointer for code point.
var pointer = indexShiftJISPointerFor(code_point)
// 8. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 9. Let lead be floor(pointer / 188).
var lead = floor(pointer / 188)
// 10. Let lead offset be 0x81, if lead is less than 0x1F, and
// 0xC1 otherwise.
var lead_offset = (lead < 0x1F) ? 0x81 : 0xC1
// 11. Let trail be pointer % 188.
var trail = pointer % 188
// 12. Let offset be 0x40, if trail is less than 0x3F, and 0x41
// otherwise.
var offset = (trail < 0x3F) ? 0x40 : 0x41
// 13. Return two bytes whose values are lead + lead offset and
// trail + offset.
return [lead + lead_offset, trail + offset]
}
}
module.exports.ShiftJISDecoder = ShiftJISDecoder
module.exports.ShiftJISEncoder = ShiftJISEncoder
+89
View File
@@ -0,0 +1,89 @@
const { end_of_stream, finished, isASCIIByte, decoderError, encoderError, isASCIICodePoint } = require('../utils');
const { indexPointerFor } = require('../indexes');
//
// 10. Legacy single-byte encodings
//
// 10.1 single-byte decoder
/**
* @implements {Decoder}
*/
class SingleByteDecoder {
/**
* @param {!Array.<number>} index The encoding index.
* @param {{fatal: boolean}} options
*/
constructor(index, options) {
const { fatal } = options
this.fatal = fatal
this.index = index
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 2. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 3. Let code point be the index code point for byte 0x80 in
// index single-byte.
var code_point = this.index[bite - 0x80]
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
}
// 10.2 single-byte encoder
/**
* @implements {Encoder}
*/
class SingleByteEncoder {
/**
* @param {!Array.<?number>} index The encoding index.
*/
constructor(index) {
this.index = index
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index pointer for code point in index
// single-byte.
const pointer = indexPointerFor(code_point, this.index)
// 4. If pointer is null, return error with code point.
if (pointer === null)
encoderError(code_point)
// 5. Return a byte whose value is pointer + 0x80.
return pointer + 0x80
}
}
module.exports.SingleByteDecoder = SingleByteDecoder
module.exports.SingleByteEncoder = SingleByteEncoder
+142
View File
@@ -0,0 +1,142 @@
const { inRange, decoderError, end_of_stream, finished, convertCodeUnitToBytes } = require('../utils');
// 15.2.1 shared utf-16 decoder
/**
* @implements {Decoder}
*/
class UTF16Decoder {
/**
* @param {boolean} utf16_be True if big-endian, false if little-endian.
* @param {{fatal: boolean}} options
*/
constructor(utf16_be, options) {
const { fatal } = options
this.utf16_be = utf16_be
this.fatal = fatal
this.utf16_lead_byte = null
this.utf16_lead_surrogate = null
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and either utf-16 lead byte or
// utf-16 lead surrogate is not null, set utf-16 lead byte and
// utf-16 lead surrogate to null, and return error.
if (bite === end_of_stream && (this.utf16_lead_byte !== null ||
this.utf16_lead_surrogate !== null)) {
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and utf-16 lead byte and utf-16
// lead surrogate are null, return finished.
if (bite === end_of_stream && this.utf16_lead_byte === null &&
this.utf16_lead_surrogate === null) {
return finished
}
// 3. If utf-16 lead byte is null, set utf-16 lead byte to byte
// and return continue.
if (this.utf16_lead_byte === null) {
this.utf16_lead_byte = bite
return null
}
// 4. Let code unit be the result of:
let code_unit
if (this.utf16_be) {
// utf-16be decoder flag is set
// (utf-16 lead byte << 8) + byte.
code_unit = (this.utf16_lead_byte << 8) + bite
} else {
// utf-16be decoder flag is unset
// (byte << 8) + utf-16 lead byte.
code_unit = (bite << 8) + this.utf16_lead_byte
}
// Then set utf-16 lead byte to null.
this.utf16_lead_byte = null
// 5. If utf-16 lead surrogate is not null, let lead surrogate
// be utf-16 lead surrogate, set utf-16 lead surrogate to null,
// and then run these substeps:
if (this.utf16_lead_surrogate !== null) {
const lead_surrogate = this.utf16_lead_surrogate
this.utf16_lead_surrogate = null
// 1. If code unit is in the range U+DC00 to U+DFFF,
// inclusive, return a code point whose value is 0x10000 +
// ((lead surrogate 0xD800) << 10) + (code unit 0xDC00).
if (inRange(code_unit, 0xDC00, 0xDFFF)) {
return 0x10000 + (lead_surrogate - 0xD800) * 0x400 +
(code_unit - 0xDC00)
}
// 2. Prepend the sequence resulting of converting code unit
// to bytes using utf-16be decoder flag to stream and return
// error.
stream.prepend(convertCodeUnitToBytes(code_unit, this.utf16_be))
return decoderError(this.fatal)
}
// 6. If code unit is in the range U+D800 to U+DBFF, inclusive,
// set utf-16 lead surrogate to code unit and return continue.
if (inRange(code_unit, 0xD800, 0xDBFF)) {
this.utf16_lead_surrogate = code_unit
return null
}
// 7. If code unit is in the range U+DC00 to U+DFFF, inclusive,
// return error.
if (inRange(code_unit, 0xDC00, 0xDFFF))
return decoderError(this.fatal)
// 8. Return code point code unit.
return code_unit
}
}
// 15.2.2 shared utf-16 encoder
/**
* @implements {Encoder}
*/
class UTF16Encoder {
/**
* @param {boolean} [utf16_be] True if big-endian, false if little-endian.
*/
constructor(utf16_be = false) {
this.utf16_be = utf16_be
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is in the range U+0000 to U+FFFF, inclusive,
// return the sequence resulting of converting code point to
// bytes using utf-16be encoder flag.
if (inRange(code_point, 0x0000, 0xFFFF))
return convertCodeUnitToBytes(code_point, this.utf16_be)
// 3. Let lead be ((code point 0x10000) >> 10) + 0xD800,
// converted to bytes using utf-16be encoder flag.
const lead = convertCodeUnitToBytes(
((code_point - 0x10000) >> 10) + 0xD800, this.utf16_be)
// 4. Let trail be ((code point 0x10000) & 0x3FF) + 0xDC00,
// converted to bytes using utf-16be encoder flag.
const trail = convertCodeUnitToBytes(
((code_point - 0x10000) & 0x3FF) + 0xDC00, this.utf16_be)
// 5. Return a byte sequence of lead followed by trail.
return lead.concat(trail)
}
}
module.exports.UTF16Decoder = UTF16Decoder
module.exports.UTF16Encoder = UTF16Encoder
+211
View File
@@ -0,0 +1,211 @@
const { inRange, decoderError, isASCIICodePoint,
end_of_stream, finished } = require('../utils');
/**
* @implements {Decoder}
*/
class UTF8Decoder {
/**
* @param {{fatal: boolean}} options
*/
constructor(options) {
const { fatal } = options
// utf-8's decoder's has an associated utf-8 code point, utf-8
// bytes seen, and utf-8 bytes needed (all initially 0), a utf-8
// lower boundary (initially 0x80), and a utf-8 upper boundary
// (initially 0xBF).
let /** @type {number} */ utf8_code_point = 0,
/** @type {number} */ utf8_bytes_seen = 0,
/** @type {number} */ utf8_bytes_needed = 0,
/** @type {number} */ utf8_lower_boundary = 0x80,
/** @type {number} */ utf8_upper_boundary = 0xBF
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
* @return {?(number|!Array.<number>)} The next code point(s)
* decoded, or null if not enough data exists in the input
* stream to decode a complete code point.
*/
this.handler = function(stream, bite) {
// 1. If byte is end-of-stream and utf-8 bytes needed is not 0,
// set utf-8 bytes needed to 0 and return error.
if (bite === end_of_stream && utf8_bytes_needed !== 0) {
utf8_bytes_needed = 0
return decoderError(fatal)
}
// 2. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 3. If utf-8 bytes needed is 0, based on byte:
if (utf8_bytes_needed === 0) {
// 0x00 to 0x7F
if (inRange(bite, 0x00, 0x7F)) {
// Return a code point whose value is byte.
return bite
}
// 0xC2 to 0xDF
else if (inRange(bite, 0xC2, 0xDF)) {
// 1. Set utf-8 bytes needed to 1.
utf8_bytes_needed = 1
// 2. Set UTF-8 code point to byte & 0x1F.
utf8_code_point = bite & 0x1F
}
// 0xE0 to 0xEF
else if (inRange(bite, 0xE0, 0xEF)) {
// 1. If byte is 0xE0, set utf-8 lower boundary to 0xA0.
if (bite === 0xE0)
utf8_lower_boundary = 0xA0
// 2. If byte is 0xED, set utf-8 upper boundary to 0x9F.
if (bite === 0xED)
utf8_upper_boundary = 0x9F
// 3. Set utf-8 bytes needed to 2.
utf8_bytes_needed = 2
// 4. Set UTF-8 code point to byte & 0xF.
utf8_code_point = bite & 0xF
}
// 0xF0 to 0xF4
else if (inRange(bite, 0xF0, 0xF4)) {
// 1. If byte is 0xF0, set utf-8 lower boundary to 0x90.
if (bite === 0xF0)
utf8_lower_boundary = 0x90
// 2. If byte is 0xF4, set utf-8 upper boundary to 0x8F.
if (bite === 0xF4)
utf8_upper_boundary = 0x8F
// 3. Set utf-8 bytes needed to 3.
utf8_bytes_needed = 3
// 4. Set UTF-8 code point to byte & 0x7.
utf8_code_point = bite & 0x7
}
// Otherwise
else {
// Return error.
return decoderError(fatal)
}
// Return continue.
return null
}
// 4. If byte is not in the range utf-8 lower boundary to utf-8
// upper boundary, inclusive, run these substeps:
if (!inRange(bite, utf8_lower_boundary, utf8_upper_boundary)) {
// 1. Set utf-8 code point, utf-8 bytes needed, and utf-8
// bytes seen to 0, set utf-8 lower boundary to 0x80, and set
// utf-8 upper boundary to 0xBF.
utf8_code_point = utf8_bytes_needed = utf8_bytes_seen = 0
utf8_lower_boundary = 0x80
utf8_upper_boundary = 0xBF
// 2. Prepend byte to stream.
stream.prepend(bite)
// 3. Return error.
return decoderError(fatal)
}
// 5. Set utf-8 lower boundary to 0x80 and utf-8 upper boundary
// to 0xBF.
utf8_lower_boundary = 0x80
utf8_upper_boundary = 0xBF
// 6. Set UTF-8 code point to (UTF-8 code point << 6) | (byte &
// 0x3F)
utf8_code_point = (utf8_code_point << 6) | (bite & 0x3F)
// 7. Increase utf-8 bytes seen by one.
utf8_bytes_seen += 1
// 8. If utf-8 bytes seen is not equal to utf-8 bytes needed,
// continue.
if (utf8_bytes_seen !== utf8_bytes_needed)
return null
// 9. Let code point be utf-8 code point.
var code_point = utf8_code_point
// 10. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes
// seen to 0.
utf8_code_point = utf8_bytes_needed = utf8_bytes_seen = 0
// 11. Return a code point whose value is code point.
return code_point
}
}
}
// 9.1.2 utf-8 encoder
/**
* @implements {Encoder}
*/
class UTF8Encoder {
constructor() {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
this.handler = function(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Set count and offset based on the range code point is in:
var count, offset
// U+0080 to U+07FF, inclusive:
if (inRange(code_point, 0x0080, 0x07FF)) {
// 1 and 0xC0
count = 1
offset = 0xC0
}
// U+0800 to U+FFFF, inclusive:
else if (inRange(code_point, 0x0800, 0xFFFF)) {
// 2 and 0xE0
count = 2
offset = 0xE0
}
// U+10000 to U+10FFFF, inclusive:
else if (inRange(code_point, 0x10000, 0x10FFFF)) {
// 3 and 0xF0
count = 3
offset = 0xF0
}
// 4. Let bytes be a byte sequence whose first byte is (code
// point >> (6 × count)) + offset.
var bytes = [(code_point >> (6 * count)) + offset]
// 5. Run these substeps while count is greater than 0:
while (count > 0) {
// 1. Set temp to code point >> (6 × (count 1)).
var temp = code_point >> (6 * (count - 1))
// 2. Append to bytes 0x80 | (temp & 0x3F).
bytes.push(0x80 | (temp & 0x3F))
// 3. Decrease count by one.
count -= 1
}
// 6. Return bytes bytes, in order.
return bytes
}
}
}
module.exports.UTF8Decoder = UTF8Decoder
module.exports.UTF8Encoder = UTF8Encoder
+59
View File
@@ -0,0 +1,59 @@
const { inRange, encoderError, end_of_stream, finished, isASCIIByte, isASCIICodePoint } = require('../utils');
// 15.5 x-user-defined
// 15.5.1 x-user-defined decoder
/**
* @implements {Decoder}
*/
class XUserDefinedDecoder {
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 2. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 3. Return a code point whose value is 0xF780 + byte 0x80.
return 0xF780 + bite - 0x80
}
}
// 15.5.2 x-user-defined encoder
/**
* @implements {Encoder}
*/
class XUserDefinedEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1.If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is in the range U+F780 to U+F7FF, inclusive,
// return a byte whose value is code point 0xF780 + 0x80.
if (inRange(code_point, 0xF780, 0xF7FF))
return code_point - 0xF780 + 0x80
// 4. Return error with code point.
return encoderError(code_point)
}
}
module.exports.XUserDefinedDecoder = XUserDefinedDecoder
module.exports.XUserDefinedEncoder = XUserDefinedEncoder
+16
View File
@@ -0,0 +1,16 @@
const TextEncoder = require('./lib/TextEncoder');
const TextDecoder = require('./lib/TextDecoder');
const EncodingIndexes = require('./encoding-indexes');
const { getEncoding } = require('./lib');
//
// Implementation of Encoding specification
// https://encoding.spec.whatwg.org/
//
module.exports.TextEncoder = TextEncoder
module.exports.TextDecoder = TextDecoder
module.exports.EncodingIndexes = EncodingIndexes
module.exports.getEncoding = getEncoding
+161
View File
@@ -0,0 +1,161 @@
const { inRange } = require('./utils');
const Indexes = require('./encoding-indexes');
//
// 6. Indexes
//
/**
* @param {number} pointer The |pointer| to search for.
* @param {(!Array.<?number>|undefined)} index The |index| to search within.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in |index|.
*/
function indexCodePointFor(pointer, i) {
if (!i) return null
return i[pointer] || null
}
/**
* @param {number} code_point The |code point| to search for.
* @param {!Array.<?number>} i The |index| to search within.
* @return {?number} The first pointer corresponding to |code point| in
* |index|, or null if |code point| is not in |index|.
*/
function indexPointerFor(code_point, i) {
var pointer = i.indexOf(code_point)
return pointer === -1 ? null : pointer
}
/**
* @param {string} name Name of the index.
*/
function index(name) {
return Indexes[name]
}
/**
* @param {number} pointer The |pointer| to search for in the gb18030 index.
* @return The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the gb18030 index.
*/
function indexGB18030RangesCodePointFor(pointer) {
// 1. If pointer is greater than 39419 and less than 189000, or
// pointer is greater than 1237575, return null.
if ((pointer > 39419 && pointer < 189000) || (pointer > 1237575))
return null
// 2. If pointer is 7457, return code point U+E7C7.
if (pointer === 7457) return 0xE7C7
// 3. Let offset be the last pointer in index gb18030 ranges that
// is equal to or less than pointer and let code point offset be
// its corresponding code point.
var offset = 0
var code_point_offset = 0
var idx = index('gb18030-ranges')
var i
for (i = 0; i < idx.length; ++i) {
/** @type {!Array.<number>} */
var entry = idx[i]
if (entry[0] <= pointer) {
offset = entry[0]
code_point_offset = entry[1]
} else {
break
}
}
// 4. Return a code point whose value is code point offset +
// pointer offset.
return code_point_offset + pointer - offset
}
/**
* @param {number} code_point The |code point| to locate in the gb18030 index.
* @return {number} The first pointer corresponding to |code point| in the
* gb18030 index.
*/
function indexGB18030RangesPointerFor(code_point) {
// 1. If code point is U+E7C7, return pointer 7457.
if (code_point === 0xE7C7) return 7457
// 2. Let offset be the last code point in index gb18030 ranges
// that is equal to or less than code point and let pointer offset
// be its corresponding pointer.
var offset = 0
var pointer_offset = 0
var idx = index('gb18030-ranges')
var i
for (i = 0; i < idx.length; ++i) {
/** @type {!Array.<number>} */
var entry = idx[i]
if (entry[1] <= code_point) {
offset = entry[1]
pointer_offset = entry[0]
} else {
break
}
}
// 3. Return a pointer whose value is pointer offset + code point
// offset.
return pointer_offset + code_point - offset
}
/**
* @param {number} code_point The |code_point| to search for in the Shift_JIS
* index.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the Shift_JIS index.
*/
function indexShiftJISPointerFor(code_point) {
// 1. Let index be index jis0208 excluding all entries whose
// pointer is in the range 8272 to 8835, inclusive.
shift_jis_index = shift_jis_index ||
index('jis0208').map((cp, pointer) => {
return inRange(pointer, 8272, 8835) ? null : cp
})
const index_ = shift_jis_index
// 2. Return the index pointer for code point in index.
return index_.indexOf(code_point)
}
var shift_jis_index
/**
* @param {number} code_point The |code_point| to search for in the big5
* index.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the big5 index.
*/
function indexBig5PointerFor(code_point) {
// 1. Let index be index Big5 excluding all entries whose pointer
big5_index_no_hkscs = big5_index_no_hkscs ||
index('big5').map((cp, pointer) => {
return (pointer < (0xA1 - 0x81) * 157) ? null : cp
})
var index_ = big5_index_no_hkscs
// 2. If code point is U+2550, U+255E, U+2561, U+256A, U+5341, or
// U+5345, return the last pointer corresponding to code point in
// index.
if (code_point === 0x2550 || code_point === 0x255E ||
code_point === 0x2561 || code_point === 0x256A ||
code_point === 0x5341 || code_point === 0x5345) {
return index_.lastIndexOf(code_point)
}
// 3. Return the index pointer for code point in index.
return indexPointerFor(code_point, index_)
}
var big5_index_no_hkscs
module.exports = index
module.exports.indexCodePointFor = indexCodePointFor
module.exports.indexPointerFor = indexPointerFor
module.exports.indexGB18030RangesCodePointFor = indexGB18030RangesCodePointFor
module.exports.indexGB18030RangesPointerFor = indexGB18030RangesPointerFor
module.exports.indexShiftJISPointerFor = indexShiftJISPointerFor
module.exports.indexBig5PointerFor = indexBig5PointerFor
+191
View File
@@ -0,0 +1,191 @@
const Stream = require('./'); const { DEFAULT_ENCODING, getEncoding } = Stream;
const { end_of_stream, finished, codePointsToString } = require('../utils');
const { decoders } = require('../table');
// 8.1 Interface TextDecoder
class TextDecoder {
/**
* @param {string=} label The label of the encoding; defaults to 'utf-8'.
* @param {Object=} options
*/
constructor(label = DEFAULT_ENCODING, options = {}) {
// A TextDecoder object has an associated encoding, decoder,
// stream, ignore BOM flag (initially unset), BOM seen flag
// (initially unset), error mode (initially replacement), and do
// not flush flag (initially unset).
/** @private */
this._encoding = null
/** @private @type {?Decoder} */
this._decoder = null
/** @private @type {boolean} */
this._ignoreBOM = false
/** @private @type {boolean} */
this._BOMseen = false
/** @private @type {string} */
this._error_mode = 'replacement'
/** @private @type {boolean} */
this._do_not_flush = false
// 1. Let encoding be the result of getting an encoding from
// label.
const encoding = getEncoding(label)
// 2. If encoding is failure or replacement, throw a RangeError.
if (encoding === null || encoding.name == 'replacement')
throw RangeError('Unknown encoding: ' + label)
if (!decoders[encoding.name]) {
throw Error('Decoder not present.' +
' Did you forget to include encoding-indexes.js first?')
}
// 4. Set dec's encoding to encoding.
this._encoding = encoding
// 5. If options's fatal member is true, set dec's error mode to
// fatal.
if (options['fatal'])
this._error_mode = 'fatal'
// 6. If options's ignoreBOM member is true, set dec's ignore BOM
// flag.
if (options['ignoreBOM'])
this._ignoreBOM = true
}
get encoding() {
return this._encoding.name.toLowerCase()
}
get fatal() {
return this._error_mode === 'fatal'
}
get ignoreBOM() {
return this._ignoreBOM
}
/**
* @param {BufferSource=} input The buffer of bytes to decode.
* @param {Object=} options
* @return The decoded string.
*/
decode(input, options = {}) {
let bytes
if (typeof input === 'object' && input instanceof ArrayBuffer) {
bytes = new Uint8Array(input)
} else if (typeof input === 'object' && 'buffer' in input &&
input.buffer instanceof ArrayBuffer) {
bytes = new Uint8Array(input.buffer,
input.byteOffset,
input.byteLength)
} else {
bytes = new Uint8Array(0)
}
// 1. If the do not flush flag is unset, set decoder to a new
// encoding's decoder, set stream to a new stream, and unset the
// BOM seen flag.
if (!this._do_not_flush) {
this._decoder = decoders[this._encoding.name]({
fatal: this._error_mode === 'fatal' })
this._BOMseen = false
}
// 2. If options's stream is true, set the do not flush flag, and
// unset the do not flush flag otherwise.
this._do_not_flush = Boolean(options['stream'])
// 3. If input is given, push a copy of input to stream.
// TODO: Align with spec algorithm - maintain stream on instance.
const input_stream = new Stream(bytes)
// 4. Let output be a new stream.
const output = []
/** @type {?(number|!Array.<number>)} */
let result
// 5. While true:
while (true) {
// 1. Let token be the result of reading from stream.
const token = input_stream.read()
// 2. If token is end-of-stream and the do not flush flag is
// set, return output, serialized.
// TODO: Align with spec algorithm.
if (token === end_of_stream)
break
// 3. Otherwise, run these subsubsteps:
// 1. Let result be the result of processing token for decoder,
// stream, output, and error mode.
result = this._decoder.handler(input_stream, token)
// 2. If result is finished, return output, serialized.
if (result === finished)
break
if (result !== null) {
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
// 3. Otherwise, if result is error, throw a TypeError.
// (Thrown in handler)
// 4. Otherwise, do nothing.
}
// TODO: Align with spec algorithm.
if (!this._do_not_flush) {
do {
result = this._decoder.handler(input_stream, input_stream.read())
if (result === finished)
break
if (result === null)
continue
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
} while (!input_stream.endOfStream())
this._decoder = null
}
return this.serializeStream(output)
}
// A TextDecoder object also has an associated serialize stream
// algorithm...
/**
* @param {!Array.<number>} stream
*/
serializeStream(stream) {
// 1. Let token be the result of reading from stream.
// (Done in-place on array, rather than as a stream)
// 2. If encoding is UTF-8, UTF-16BE, or UTF-16LE, and ignore
// BOM flag and BOM seen flag are unset, run these subsubsteps:
if (['UTF-8', 'UTF-16LE', 'UTF-16BE'].includes(this._encoding.name) &&
!this._ignoreBOM && !this._BOMseen) {
if (stream.length > 0 && stream[0] === 0xFEFF) {
// 1. If token is U+FEFF, set BOM seen flag.
this._BOMseen = true
stream.shift()
} else if (stream.length > 0) {
// 2. Otherwise, if token is not end-of-stream, set BOM seen
// flag and append token to stream.
this._BOMseen = true
} else {
// 3. Otherwise, if token is not end-of-stream, append token
// to output.
// (no-op)
}
}
// 4. Otherwise, return output.
return codePointsToString(stream)
}
}
module.exports = TextDecoder
+108
View File
@@ -0,0 +1,108 @@
const Stream = require('./'); const { DEFAULT_ENCODING, getEncoding } = Stream;
const { end_of_stream, finished, stringToCodePoints } = require('../utils');
const { encoders } = require('../table');
// 8.2 Interface TextEncoder
class TextEncoder {
/**
* @param {string=} label The label of the encoding. NONSTANDARD.
* @param {Object=} [options] NONSTANDARD.
*/
constructor(label, options = {}) {
// A TextEncoder object has an associated encoding and encoder.
/** @private */
this._encoding = null
/** @private @type {?Encoder} */
this._encoder = null
// Non-standard
/** @private @type {boolean} */
this._do_not_flush = false
/** @private @type {string} */
this._fatal = options['fatal'] ? 'fatal' : 'replacement'
// 2. Set enc's encoding to UTF-8's encoder.
if (options['NONSTANDARD_allowLegacyEncoding']) {
// NONSTANDARD behavior.
label = label !== undefined ? String(label) : DEFAULT_ENCODING
var encoding = getEncoding(label)
if (encoding === null || encoding.name === 'replacement')
throw RangeError('Unknown encoding: ' + label)
if (!encoders[encoding.name]) {
throw Error('Encoder not present.' +
' Did you forget to include encoding-indexes.js first?')
}
this._encoding = encoding
} else {
// Standard behavior.
this._encoding = getEncoding('utf-8')
if (label !== undefined && 'console' in global) {
console.warn('TextEncoder constructor called with encoding label, '
+ 'which is ignored.')
}
}
}
get encoding() {
return this._encoding.name.toLowerCase()
}
/**
* @param {string=} opt_string The string to encode.
* @param {Object=} options
*/
encode(opt_string = '', options = {}) {
// NOTE: This option is nonstandard. None of the encodings
// permitted for encoding (i.e. UTF-8, UTF-16) are stateful when
// the input is a USVString so streaming is not necessary.
if (!this._do_not_flush)
this._encoder = encoders[this._encoding.name]({
fatal: this._fatal === 'fatal' })
this._do_not_flush = Boolean(options['stream'])
// 1. Convert input to a stream.
const input = new Stream(stringToCodePoints(opt_string))
// 2. Let output be a new stream
const output = []
/** @type {?(number|!Array.<number>)} */
var result
// 3. While true, run these substeps:
while (true) {
// 1. Let token be the result of reading from input.
var token = input.read()
if (token === end_of_stream)
break
// 2. Let result be the result of processing token for encoder,
// input, output.
result = this._encoder.handler(input, token)
if (result === finished)
break
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
// TODO: Align with spec algorithm.
if (!this._do_not_flush) {
while (true) {
result = this._encoder.handler(input, input.read())
if (result === finished)
break
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
this._encoder = null
}
// 3. If result is finished, convert output into a byte sequence,
// and then return a Uint8Array object wrapping an ArrayBuffer
// containing output.
return new Uint8Array(output)
}
}
module.exports = TextEncoder
+122
View File
@@ -0,0 +1,122 @@
const { end_of_stream } = require('../utils');
const { label_to_encoding } = require('../table');
class Stream {
/**
* A stream represents an ordered sequence of tokens.
* @param {!(Array.<number>|Uint8Array)} tokens Array of tokens that provide
* the stream.
*/
constructor(tokens) {
this.tokens = [...tokens]
// Reversed as push/pop is more efficient than shift/unshift.
this.tokens.reverse()
}
/**
* @returns True if end-of-stream has been hit.
*/
endOfStream() {
return !this.tokens.length
}
/**
* When a token is read from a stream, the first token in the
* stream must be returned and subsequently removed, and
* end-of-stream must be returned otherwise.
*
* @return Get the next token from the stream, or end_of_stream.
*/
read() {
if (!this.tokens.length)
return end_of_stream
return this.tokens.pop()
}
/**
* When one or more tokens are prepended to a stream, those tokens
* must be inserted, in given order, before the first token in the
* stream.
*
* @param {(number|!Array.<number>)} token The token(s) to prepend to the
* stream.
*/
prepend(token) {
if (Array.isArray(token)) {
var tokens = /**@type {!Array.<number>}*/(token)
while (tokens.length)
this.tokens.push(tokens.pop())
} else {
this.tokens.push(token)
}
}
/**
* When one or more tokens are pushed to a stream, those tokens
* must be inserted, in given order, after the last token in the
* stream.
*
* @param {(number|!Array.<number>)} token The tokens(s) to push to the
* stream.
*/
push(token) {
if (Array.isArray(token)) {
const tokens = /**@type {!Array.<number>}*/(token)
while (tokens.length)
this.tokens.unshift(tokens.shift())
} else {
this.tokens.unshift(token)
}
}
}
const DEFAULT_ENCODING = 'utf-8'
/**
* Returns the encoding for the label.
* @param {string} label The encoding label.
*/
function getEncoding(label) {
// 1. Remove any leading and trailing ASCII whitespace from label.
label = String(label).trim().toLowerCase()
// 2. If label is an ASCII case-insensitive match for any of the
// labels listed in the table below, return the corresponding
// encoding, and failure otherwise.
if (Object.prototype.hasOwnProperty.call(label_to_encoding, label)) {
return label_to_encoding[label]
}
return null
}
//
// 5. Encodings
//
// 5.1 Encoders and decoders
// /** @interface */
// function Decoder() {}
// Decoder.prototype = {
// /**
// * @param {Stream} stream The stream of bytes being decoded.
// * @param {number} bite The next byte read from the stream.
// * @return {?(number|!Array.<number>)} The next code point(s)
// * decoded, or null if not enough data exists in the input
// * stream to decode a complete code point, or |finished|.
// */
// handler: function(stream, bite) {},
// }
// /** @interface */
// function Encoder() {}
// Encoder.prototype = {
// /**
// * @param {Stream} stream The stream of code points being encoded.
// * @param {number} code_point Next code point read from the stream.
// * @return {(number|!Array.<number>)} Byte(s) to emit, or |finished|.
// */
// handler: function(stream, code_point) {},
// }
module.exports = Stream
module.exports.DEFAULT_ENCODING = DEFAULT_ENCODING
module.exports.getEncoding = getEncoding
+123
View File
@@ -0,0 +1,123 @@
const Encodings = require('./encodings');
const { UTF8Decoder, UTF8Encoder } = require('./implementations/utf8');
const { UTF16Decoder, UTF16Encoder } = require('./implementations/utf16');
const { GB18030Decoder, GB18030Encoder } = require('./implementations/gb18030');
const { Big5Decoder, Big5Encoder } = require('./implementations/big5');
const { EUCJPDecoder, EUCJPEncoder } = require('./implementations/euc-jp');
const { EUCKRDecoder, EUCKREncoder } = require('./implementations/euc-kr');
const { ISO2022JPDecoder, ISO2022JPEncoder } = require('./implementations/iso-2022-jp');
const { XUserDefinedDecoder, XUserDefinedEncoder } = require('./implementations/x-user-defined');
const { ShiftJISDecoder, ShiftJISEncoder } = require('./implementations/shift-jis');
const { SingleByteDecoder, SingleByteEncoder } = require('./implementations/single-byte');
const index = require('./indexes');;
// 5.2 Names and labels
// TODO: Define @typedef for Encoding: {name:string,labels:Array.<string>}
// https://github.com/google/closure-compiler/issues/247
// Label to encoding registry.
/** @type {Object.<string,{name:string,labels:Array.<string>}>} */
const label_to_encoding = {}
Encodings.forEach(({ encodings }) => {
encodings.forEach((encoding) => {
encoding.labels.forEach((label) => {
label_to_encoding[label] = encoding
})
})
})
// Registry of of encoder/decoder factories, by encoding name.
const encoders = {
'UTF-8'() { // 9.1 utf-8
return new UTF8Encoder()
},
'GBK'(options) { // 11.1.2 gbk encoder;
// gbk's encoder is gb18030's encoder with its gbk flag set.
return new GB18030Encoder(options, true)
},
'gb18030'() {
return new GB18030Encoder()
},
'Big5'() {
return new Big5Encoder()
},
'EUC-JP'() {
return new EUCJPEncoder()
},
'EUC-KR'() {
return new EUCKREncoder()
},
'ISO-2022-JP'() {
return new ISO2022JPEncoder()
},
'UTF-16BE'() { // 15.3 utf-16be
return new UTF16Encoder(true)
},
'UTF-16LE'() { // 15.4 utf-16le
return new UTF16Encoder()
},
'x-user-defined'() {
return new XUserDefinedEncoder()
},
'Shift_JIS'() {
return new ShiftJISEncoder()
},
}
/** @type {Object.<string, function({fatal:boolean}): Decoder>} */
const decoders = {
'UTF-8'(options) { // 9.1.1 utf-8 decoder
return new UTF8Decoder(options)
},
'GBK'(options) { // 11.1.1 gbk decoder; gbk's decoder is gb18030's decoder.
return new GB18030Decoder(options)
},
'gb18030'(options) {
return new GB18030Decoder(options)
},
'Big5'(options) {
return new Big5Decoder(options)
},
'EUC-JP'(options) {
return new EUCJPDecoder(options)
},
'EUC-KR'(options) {
return new EUCKRDecoder(options)
},
'ISO-2022-JP'(options) {
return new ISO2022JPDecoder(options)
},
'UTF-16BE'(options) { // 15.3.1 utf-16be decoder
return new UTF16Decoder(true, options)
},
'UTF-16LE'(options) { // 15.4.1 utf-16le decoder
return new UTF16Decoder(false, options)
},
'x-user-defined'() {
return new XUserDefinedDecoder()
},
'Shift_JIS'(options) {
return new ShiftJISDecoder(options)
},
}
Encodings.forEach(({ heading, encodings }) => {
if (heading != 'Legacy single-byte encodings')
return
encodings.forEach((encoding) => {
const name = encoding.name
const idx = index(name.toLowerCase())
decoders[name] = (options) => {
return new SingleByteDecoder(idx, options)
}
encoders[name] = (options) => {
return new SingleByteEncoder(idx, options)
}
})
})
module.exports.label_to_encoding = label_to_encoding
module.exports.encoders = encoders
module.exports.decoders = decoders
+192
View File
@@ -0,0 +1,192 @@
//
// Utilities
//
/**
* @param {number} a The number to test.
* @param {number} min The minimum value in the range, inclusive.
* @param {number} max The maximum value in the range, inclusive.
* @return {boolean} True if a >= min and a <= max.
*/
function inRange(a, min, max) {
return min <= a && a <= max
}
const floor = Math.floor
/**
* @param {string} string Input string of UTF-16 code units.
* @return {!Array.<number>} Code points.
*/
function stringToCodePoints(string) {
// https://heycam.github.io/webidl/#dfn-obtain-unicode
// 1. Let S be the DOMString value.
var s = String(string)
// 2. Let n be the length of S.
var n = s.length
// 3. Initialize i to 0.
var i = 0
// 4. Initialize U to be an empty sequence of Unicode characters.
var u = []
// 5. While i < n:
while (i < n) {
// 1. Let c be the code unit in S at index i.
var c = s.charCodeAt(i)
// 2. Depending on the value of c:
// c < 0xD800 or c > 0xDFFF
if (c < 0xD800 || c > 0xDFFF) {
// Append to U the Unicode character with code point c.
u.push(c)
}
// 0xDC00 ≤ c ≤ 0xDFFF
else if (0xDC00 <= c && c <= 0xDFFF) {
// Append to U a U+FFFD REPLACEMENT CHARACTER.
u.push(0xFFFD)
}
// 0xD800 ≤ c ≤ 0xDBFF
else if (0xD800 <= c && c <= 0xDBFF) {
// 1. If i = n1, then append to U a U+FFFD REPLACEMENT
// CHARACTER.
if (i === n - 1) {
u.push(0xFFFD)
}
// 2. Otherwise, i < n1:
else {
// 1. Let d be the code unit in S at index i+1.
var d = s.charCodeAt(i + 1)
// 2. If 0xDC00 ≤ d ≤ 0xDFFF, then:
if (0xDC00 <= d && d <= 0xDFFF) {
// 1. Let a be c & 0x3FF.
var a = c & 0x3FF
// 2. Let b be d & 0x3FF.
var b = d & 0x3FF
// 3. Append to U the Unicode character with code point
// 2^16+2^10*a+b.
u.push(0x10000 + (a << 10) + b)
// 4. Set i to i+1.
i += 1
}
// 3. Otherwise, d < 0xDC00 or d > 0xDFFF. Append to U a
// U+FFFD REPLACEMENT CHARACTER.
else {
u.push(0xFFFD)
}
}
}
// 3. Set i to i+1.
i += 1
}
// 6. Return U.
return u
}
/**
* @param {!Array.<number>} code_points Array of code points.
* @return {string} string String of UTF-16 code units.
*/
function codePointsToString(code_points) {
var s = ''
for (var i = 0; i < code_points.length; ++i) {
var cp = code_points[i]
if (cp <= 0xFFFF) {
s += String.fromCharCode(cp)
} else {
cp -= 0x10000
s += String.fromCharCode((cp >> 10) + 0xD800,
(cp & 0x3FF) + 0xDC00)
}
}
return s
}
/**
* @param {boolean} fatal If true, decoding errors raise an exception.
* @param {number=} opt_code_point Override the standard fallback code point.
* @return The code point to insert on a decoding error.
*/
function decoderError(fatal, opt_code_point) {
if (fatal)
throw TypeError('Decoder error')
return opt_code_point || 0xFFFD
}
/**
* @param {number} code_point The code point that could not be encoded.
* @return {number} Always throws, no value is actually returned.
*/
function encoderError(code_point) {
throw TypeError('The code point ' + code_point + ' could not be encoded.')
}
/**
* @param {number} code_unit
* @param {boolean} utf16be
*/
function convertCodeUnitToBytes(code_unit, utf16be) {
// 1. Let byte1 be code unit >> 8.
const byte1 = code_unit >> 8
// 2. Let byte2 be code unit & 0x00FF.
const byte2 = code_unit & 0x00FF
// 3. Then return the bytes in order:
// utf-16be flag is set: byte1, then byte2.
if (utf16be)
return [byte1, byte2]
// utf-16be flag is unset: byte2, then byte1.
return [byte2, byte1]
}
//
// 4. Terminology
//
/**
* An ASCII byte is a byte in the range 0x00 to 0x7F, inclusive.
* @param {number} a The number to test.
* @return {boolean} True if a is in the range 0x00 to 0x7F, inclusive.
*/
function isASCIIByte(a) {
return 0x00 <= a && a <= 0x7F
}
/**
* An ASCII code point is a code point in the range U+0000 to
* U+007F, inclusive.
*/
const isASCIICodePoint = isASCIIByte
/**
* End-of-stream is a special token that signifies no more tokens are in the stream.
*/
const end_of_stream = -1
const finished = -1
module.exports.inRange = inRange
module.exports.floor = floor
module.exports.stringToCodePoints = stringToCodePoints
module.exports.codePointsToString = codePointsToString
module.exports.decoderError = decoderError
module.exports.encoderError = encoderError
module.exports.convertCodeUnitToBytes = convertCodeUnitToBytes
module.exports.isASCIIByte = isASCIIByte
module.exports.isASCIICodePoint = isASCIICodePoint
module.exports.end_of_stream = end_of_stream
module.exports.finished = finished
+66
View File
@@ -0,0 +1,66 @@
{
"name": "text-decoding",
"version": "1.0.0",
"description": "[fork] TextEncoder and TextDecoder (Polyfill for the Encoding Living Standard's API) For Node.JS.",
"main": "build/index.js",
"module": "src/index.js",
"scripts": {
"t": "zoroaster -a",
"test": "yarn t test/spec test/mask",
"spec": "yarn t test/spec",
"mask": "yarn t test/mask",
"test-build": "ALAMODE_ENV=test-build yarn test",
"lint": "eslint .",
"doc": "NODE_DEBUG=doc doc -o README.md",
"b": "alamode src -o build -s",
"d": "yarn-s d1 externs",
"d1": "typal types/index.js src -c -t types/index.xml",
"externs": "typal types/externs.js",
"build": "yarn-s d b doc",
"e": "alanode"
},
"files": [
"build",
"src",
"types/externs.js"
],
"externs": "types/externs.js",
"repository": {
"type": "git",
"url": "git://github.com/idiocc/text-decoding.git"
},
"keywords": [
"text-encoding",
"encoding",
"text-decoding",
"TextEncoder",
"TextDecoder",
"decoding",
"ascii",
"windows-1252",
"big5",
"euc-jp",
"euc-kr",
"gb18030",
"iso-20220jp",
"shift-jis",
"single-byte",
"utf8",
"utf16",
"x-user-defined",
"unicode"
],
"author": "Anton <anton@adc.sh>",
"license": "MIT",
"bugs": {
"url": "https://github.com/idiocc/text-decoding/issues"
},
"homepage": "https://github.com/idiocc/text-decoding#readme",
"devDependencies": {
"alamode": "^2.3.4",
"documentary": "^1.27.4",
"eslint-config-artdeco": "1.0.1",
"yarn-s": "1.1.0",
"zoroaster": "^4.1.1-alpha"
}
}
File diff suppressed because one or more lines are too long
+460
View File
@@ -0,0 +1,460 @@
/**
* Encodings table: https://encoding.spec.whatwg.org/encodings.json
*/
const encodings = [
{
encodings: [
{
labels: [
"unicode-1-1-utf-8",
"utf-8",
"utf8",
],
name: "UTF-8",
},
],
heading: "The Encoding",
},
{
encodings: [
{
labels: [
"866",
"cp866",
"csibm866",
"ibm866",
],
name: "IBM866",
},
{
labels: [
"csisolatin2",
"iso-8859-2",
"iso-ir-101",
"iso8859-2",
"iso88592",
"iso_8859-2",
"iso_8859-2:1987",
"l2",
"latin2",
],
name: "ISO-8859-2",
},
{
labels: [
"csisolatin3",
"iso-8859-3",
"iso-ir-109",
"iso8859-3",
"iso88593",
"iso_8859-3",
"iso_8859-3:1988",
"l3",
"latin3",
],
name: "ISO-8859-3",
},
{
labels: [
"csisolatin4",
"iso-8859-4",
"iso-ir-110",
"iso8859-4",
"iso88594",
"iso_8859-4",
"iso_8859-4:1988",
"l4",
"latin4",
],
name: "ISO-8859-4",
},
{
labels: [
"csisolatincyrillic",
"cyrillic",
"iso-8859-5",
"iso-ir-144",
"iso8859-5",
"iso88595",
"iso_8859-5",
"iso_8859-5:1988",
],
name: "ISO-8859-5",
},
{
labels: [
"arabic",
"asmo-708",
"csiso88596e",
"csiso88596i",
"csisolatinarabic",
"ecma-114",
"iso-8859-6",
"iso-8859-6-e",
"iso-8859-6-i",
"iso-ir-127",
"iso8859-6",
"iso88596",
"iso_8859-6",
"iso_8859-6:1987",
],
name: "ISO-8859-6",
},
{
labels: [
"csisolatingreek",
"ecma-118",
"elot_928",
"greek",
"greek8",
"iso-8859-7",
"iso-ir-126",
"iso8859-7",
"iso88597",
"iso_8859-7",
"iso_8859-7:1987",
"sun_eu_greek",
],
name: "ISO-8859-7",
},
{
labels: [
"csiso88598e",
"csisolatinhebrew",
"hebrew",
"iso-8859-8",
"iso-8859-8-e",
"iso-ir-138",
"iso8859-8",
"iso88598",
"iso_8859-8",
"iso_8859-8:1988",
"visual",
],
name: "ISO-8859-8",
},
{
labels: [
"csiso88598i",
"iso-8859-8-i",
"logical",
],
name: "ISO-8859-8-I",
},
{
labels: [
"csisolatin6",
"iso-8859-10",
"iso-ir-157",
"iso8859-10",
"iso885910",
"l6",
"latin6",
],
name: "ISO-8859-10",
},
{
labels: [
"iso-8859-13",
"iso8859-13",
"iso885913",
],
name: "ISO-8859-13",
},
{
labels: [
"iso-8859-14",
"iso8859-14",
"iso885914",
],
name: "ISO-8859-14",
},
{
labels: [
"csisolatin9",
"iso-8859-15",
"iso8859-15",
"iso885915",
"iso_8859-15",
"l9",
],
name: "ISO-8859-15",
},
{
labels: [
"iso-8859-16",
],
name: "ISO-8859-16",
},
{
labels: [
"cskoi8r",
"koi",
"koi8",
"koi8-r",
"koi8_r",
],
name: "KOI8-R",
},
{
labels: [
"koi8-ru",
"koi8-u",
],
name: "KOI8-U",
},
{
labels: [
"csmacintosh",
"mac",
"macintosh",
"x-mac-roman",
],
name: "macintosh",
},
{
labels: [
"dos-874",
"iso-8859-11",
"iso8859-11",
"iso885911",
"tis-620",
"windows-874",
],
name: "windows-874",
},
{
labels: [
"cp1250",
"windows-1250",
"x-cp1250",
],
name: "windows-1250",
},
{
labels: [
"cp1251",
"windows-1251",
"x-cp1251",
],
name: "windows-1251",
},
{
labels: [
"ansi_x3.4-1968",
"ascii",
"cp1252",
"cp819",
"csisolatin1",
"ibm819",
"iso-8859-1",
"iso-ir-100",
"iso8859-1",
"iso88591",
"iso_8859-1",
"iso_8859-1:1987",
"l1",
"latin1",
"us-ascii",
"windows-1252",
"x-cp1252",
],
name: "windows-1252",
},
{
labels: [
"cp1253",
"windows-1253",
"x-cp1253",
],
name: "windows-1253",
},
{
labels: [
"cp1254",
"csisolatin5",
"iso-8859-9",
"iso-ir-148",
"iso8859-9",
"iso88599",
"iso_8859-9",
"iso_8859-9:1989",
"l5",
"latin5",
"windows-1254",
"x-cp1254",
],
name: "windows-1254",
},
{
labels: [
"cp1255",
"windows-1255",
"x-cp1255",
],
name: "windows-1255",
},
{
labels: [
"cp1256",
"windows-1256",
"x-cp1256",
],
name: "windows-1256",
},
{
labels: [
"cp1257",
"windows-1257",
"x-cp1257",
],
name: "windows-1257",
},
{
labels: [
"cp1258",
"windows-1258",
"x-cp1258",
],
name: "windows-1258",
},
{
labels: [
"x-mac-cyrillic",
"x-mac-ukrainian",
],
name: "x-mac-cyrillic",
},
],
heading: "Legacy single-byte encodings",
},
{
encodings: [
{
labels: [
"chinese",
"csgb2312",
"csiso58gb231280",
"gb2312",
"gb_2312",
"gb_2312-80",
"gbk",
"iso-ir-58",
"x-gbk",
],
name: "GBK",
},
{
labels: [
"gb18030",
],
name: "gb18030",
},
],
heading: "Legacy multi-byte Chinese (simplified) encodings",
},
{
encodings: [
{
labels: [
"big5",
"big5-hkscs",
"cn-big5",
"csbig5",
"x-x-big5",
],
name: "Big5",
},
],
heading: "Legacy multi-byte Chinese (traditional) encodings",
},
{
encodings: [
{
labels: [
"cseucpkdfmtjapanese",
"euc-jp",
"x-euc-jp",
],
name: "EUC-JP",
},
{
labels: [
"csiso2022jp",
"iso-2022-jp",
],
name: "ISO-2022-JP",
},
{
labels: [
"csshiftjis",
"ms932",
"ms_kanji",
"shift-jis",
"shift_jis",
"sjis",
"windows-31j",
"x-sjis",
],
name: "Shift_JIS",
},
],
heading: "Legacy multi-byte Japanese encodings",
},
{
encodings: [
{
labels: [
"cseuckr",
"csksc56011987",
"euc-kr",
"iso-ir-149",
"korean",
"ks_c_5601-1987",
"ks_c_5601-1989",
"ksc5601",
"ksc_5601",
"windows-949",
],
name: "EUC-KR",
},
],
heading: "Legacy multi-byte Korean encodings",
},
{
encodings: [
{
labels: [
"csiso2022kr",
"hz-gb-2312",
"iso-2022-cn",
"iso-2022-cn-ext",
"iso-2022-kr",
],
name: "replacement",
},
{
labels: [
"utf-16be",
],
name: "UTF-16BE",
},
{
labels: [
"utf-16",
"utf-16le",
],
name: "UTF-16LE",
},
{
labels: [
"x-user-defined",
],
name: "x-user-defined",
},
],
heading: "Legacy miscellaneous encodings",
},
]
export default encodings
+154
View File
@@ -0,0 +1,154 @@
import { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } from '../utils'
import index, { indexBig5PointerFor, indexCodePointFor } from '../indexes'
//
// 12. Legacy multi-byte Chinese (traditional) encodings
//
// 12.1 Big5
// 12.1.1 Big5 decoder
/**
* @implements {Decoder}
*/
export class Big5Decoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// Big5's decoder has an associated Big5 lead (initially 0x00).
this.Big5_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and Big5 lead is not 0x00, set
// Big5 lead to 0x00 and return error.
if (bite === end_of_stream && this.Big5_lead !== 0x00) {
this.Big5_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and Big5 lead is 0x00, return
// finished.
if (bite === end_of_stream && this.Big5_lead === 0x00)
return finished
// 3. If Big5 lead is not 0x00, let lead be Big5 lead, let
// pointer be null, set Big5 lead to 0x00, and then run these
// substeps:
if (this.Big5_lead !== 0x00) {
const lead = this.Big5_lead
let pointer = null
this.Big5_lead = 0x00
// 1. Let offset be 0x40 if byte is less than 0x7F and 0x62
// otherwise.
const offset = bite < 0x7F ? 0x40 : 0x62
// 2. If byte is in the range 0x40 to 0x7E, inclusive, or 0xA1
// to 0xFE, inclusive, set pointer to (lead 0x81) × 157 +
// (byte offset).
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0xA1, 0xFE))
pointer = (lead - 0x81) * 157 + (bite - offset)
// 3. If there is a row in the table below whose first column
// is pointer, return the two code points listed in its second
// column
// Pointer | Code points
// --------+--------------
// 1133 | U+00CA U+0304
// 1135 | U+00CA U+030C
// 1164 | U+00EA U+0304
// 1166 | U+00EA U+030C
switch (pointer) {
case 1133: return [0x00CA, 0x0304]
case 1135: return [0x00CA, 0x030C]
case 1164: return [0x00EA, 0x0304]
case 1166: return [0x00EA, 0x030C]
}
// 4. Let code point be null if pointer is null and the index
// code point for pointer in index Big5 otherwise.
const code_point = (pointer === null) ? null :
indexCodePointFor(pointer, index('big5'))
// 5. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 6. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 7. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 5. If byte is in the range 0x81 to 0xFE, inclusive, set Big5
// lead to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.Big5_lead = bite
return null
}
// 6. Return error.
return decoderError(this.fatal)
}
}
// 12.1.2 Big5 encoder
/**
* @implements {Encoder}
*/
export class Big5Encoder {
constructor() {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
this.handler = function(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index Big5 pointer for code point.
const pointer = indexBig5PointerFor(code_point)
// 4. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 5. Let lead be floor(pointer / 157) + 0x81.
const lead = floor(pointer / 157) + 0x81
// 6. If lead is less than 0xA1, return error with code point.
if (lead < 0xA1)
return encoderError(code_point)
// 7. Let trail be pointer % 157.
const trail = pointer % 157
// 8. Let offset be 0x40 if trail is less than 0x3F and 0x62
// otherwise.
const offset = trail < 0x3F ? 0x40 : 0x62
// Return two bytes whose values are lead and trail + offset.
return [lead, trail + offset]
}
}
}
+166
View File
@@ -0,0 +1,166 @@
import { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } from '../utils'
import index, { indexCodePointFor, indexPointerFor } from '../indexes'
//
// 13. Legacy multi-byte Japanese encodings
//
// 13.1 euc-jp
// 13.1.1 euc-jp decoder
/**
* @implements {Decoder}
*/
export class EUCJPDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// euc-jp's decoder has an associated euc-jp jis0212 flag
// (initially unset) and euc-jp lead (initially 0x00).
this.eucjp_jis0212_flag = false
this.eucjp_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and euc-jp lead is not 0x00, set
// euc-jp lead to 0x00, and return error.
if (bite === end_of_stream && this.eucjp_lead !== 0x00) {
this.eucjp_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and euc-jp lead is 0x00, return
// finished.
if (bite === end_of_stream && this.eucjp_lead === 0x00)
return finished
// 3. If euc-jp lead is 0x8E and byte is in the range 0xA1 to
// 0xDF, inclusive, set euc-jp lead to 0x00 and return a code
// point whose value is 0xFF61 0xA1 + byte.
if (this.eucjp_lead === 0x8E && inRange(bite, 0xA1, 0xDF)) {
this.eucjp_lead = 0x00
return 0xFF61 - 0xA1 + bite
}
// 4. If euc-jp lead is 0x8F and byte is in the range 0xA1 to
// 0xFE, inclusive, set the euc-jp jis0212 flag, set euc-jp lead
// to byte, and return continue.
if (this.eucjp_lead === 0x8F && inRange(bite, 0xA1, 0xFE)) {
this.eucjp_jis0212_flag = true
this.eucjp_lead = bite
return null
}
// 5. If euc-jp lead is not 0x00, let lead be euc-jp lead, set
// euc-jp lead to 0x00, and run these substeps:
if (this.eucjp_lead !== 0x00) {
const lead = this.eucjp_lead
this.eucjp_lead = 0x00
// 1. Let code point be null.
let code_point = null
// 2. If lead and byte are both in the range 0xA1 to 0xFE,
// inclusive, set code point to the index code point for (lead
// 0xA1) × 94 + byte 0xA1 in index jis0208 if the euc-jp
// jis0212 flag is unset and in index jis0212 otherwise.
if (inRange(lead, 0xA1, 0xFE) && inRange(bite, 0xA1, 0xFE)) {
code_point = indexCodePointFor(
(lead - 0xA1) * 94 + (bite - 0xA1),
index(!this.eucjp_jis0212_flag ? 'jis0208' : 'jis0212'))
}
// 3. Unset the euc-jp jis0212 flag.
this.eucjp_jis0212_flag = false
// 4. If byte is not in the range 0xA1 to 0xFE, inclusive,
// prepend byte to stream.
if (!inRange(bite, 0xA1, 0xFE))
stream.prepend(bite)
// 5. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 6. Return a code point whose value is code point.
return code_point
}
// 6. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 7. If byte is 0x8E, 0x8F, or in the range 0xA1 to 0xFE,
// inclusive, set euc-jp lead to byte and return continue.
if (bite === 0x8E || bite === 0x8F || inRange(bite, 0xA1, 0xFE)) {
this.eucjp_lead = bite
return null
}
// 8. Return error.
return decoderError(this.fatal)
}
}
// 13.1.2 euc-jp encoder
/**
* @implements {Encoder}
*/
export class EUCJPEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 4. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
// 5. If code point is in the range U+FF61 to U+FF9F, inclusive,
// return two bytes whose values are 0x8E and code point
// 0xFF61 + 0xA1.
if (inRange(code_point, 0xFF61, 0xFF9F))
return [0x8E, code_point - 0xFF61 + 0xA1]
// 6. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 7. Let pointer be the index pointer for code point in index
// jis0208.
const pointer = indexPointerFor(code_point, index('jis0208'))
// 8. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 9. Let lead be floor(pointer / 94) + 0xA1.
const lead = floor(pointer / 94) + 0xA1
// 10. Let trail be pointer % 94 + 0xA1.
const trail = pointer % 94 + 0xA1
// 11. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
+124
View File
@@ -0,0 +1,124 @@
import { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } from '../utils'
import index, { indexCodePointFor, indexPointerFor } from '../indexes'
//
// 14. Legacy multi-byte Korean encodings
//
// 14.1 euc-kr
// 14.1.1 euc-kr decoder
/**
* @implements {Decoder}
*/
export class EUCKRDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// euc-kr's decoder has an associated euc-kr lead (initially 0x00).
this.euckr_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and euc-kr lead is not 0x00, set
// euc-kr lead to 0x00 and return error.
if (bite === end_of_stream && this.euckr_lead !== 0) {
this.euckr_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and euc-kr lead is 0x00, return
// finished.
if (bite === end_of_stream && this.euckr_lead === 0)
return finished
// 3. If euc-kr lead is not 0x00, let lead be euc-kr lead, let
// pointer be null, set euc-kr lead to 0x00, and then run these
// substeps:
if (this.euckr_lead !== 0x00) {
const lead = this.euckr_lead
let pointer = null
this.euckr_lead = 0x00
// 1. If byte is in the range 0x41 to 0xFE, inclusive, set
// pointer to (lead 0x81) × 190 + (byte 0x41).
if (inRange(bite, 0x41, 0xFE))
pointer = (lead - 0x81) * 190 + (bite - 0x41)
// 2. Let code point be null, if pointer is null, and the
// index code point for pointer in index euc-kr otherwise.
const code_point = (pointer === null)
? null : indexCodePointFor(pointer, index('euc-kr'))
// 3. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (pointer === null && isASCIIByte(bite))
stream.prepend(bite)
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 5. If byte is in the range 0x81 to 0xFE, inclusive, set
// euc-kr lead to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.euckr_lead = bite
return null
}
// 6. Return error.
return decoderError(this.fatal)
}
}
// 14.1.2 euc-kr encoder
/**
* @implements {Encoder}
*/
export class EUCKREncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index pointer for code point in index
// euc-kr.
const pointer = indexPointerFor(code_point, index('euc-kr'))
// 4. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 5. Let lead be floor(pointer / 190) + 0x81.
const lead = floor(pointer / 190) + 0x81
// 6. Let trail be pointer % 190 + 0x41.
const trail = (pointer % 190) + 0x41
// 7. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
+251
View File
@@ -0,0 +1,251 @@
import { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, isASCIIByte, floor } from '../utils'
import index, {
indexGB18030RangesCodePointFor, indexGB18030RangesPointerFor,
indexCodePointFor, indexPointerFor } from '../indexes'
// 11.2 gb18030
// 11.2.1 gb18030 decoder
/**
* @constructor
* @implements {Decoder}
* @param {{fatal: boolean}} options
*/
export class GB18030Decoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// gb18030's decoder has an associated gb18030 first, gb18030
// second, and gb18030 third (all initially 0x00).
this.gb18030_first = 0x00
this.gb18030_second = 0x00,
this.gb18030_third = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
* @return The next code point(s) decoded, or null if not enough data exists in the input stream to decode a complete code point.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and gb18030 first, gb18030
// second, and gb18030 third are 0x00, return finished.
if (bite === end_of_stream && this.gb18030_first === 0x00 &&
this.gb18030_second === 0x00 && this.gb18030_third === 0x00) {
return finished
}
// 2. If byte is end-of-stream, and gb18030 first, gb18030
// second, or gb18030 third is not 0x00, set gb18030 first,
// gb18030 second, and gb18030 third to 0x00, and return error.
if (bite === end_of_stream &&
(this.gb18030_first !== 0x00 || this.gb18030_second !== 0x00 ||
this.gb18030_third !== 0x00)) {
this.gb18030_first = 0x00
this.gb18030_second = 0x00
this.gb18030_third = 0x00
decoderError(this.fatal)
}
var code_point
// 3. If gb18030 third is not 0x00, run these substeps:
if (this.gb18030_third !== 0x00) {
// 1. Let code point be null.
code_point = null
// 2. If byte is in the range 0x30 to 0x39, inclusive, set
// code point to the index gb18030 ranges code point for
// (((gb18030 first 0x81) × 10 + gb18030 second 0x30) ×
// 126 + gb18030 third 0x81) × 10 + byte 0x30.
if (inRange(bite, 0x30, 0x39)) {
code_point = indexGB18030RangesCodePointFor(
(((this.gb18030_first - 0x81) * 10 + this.gb18030_second - 0x30) * 126 +
this.gb18030_third - 0x81) * 10 + bite - 0x30)
}
// 3. Let buffer be a byte sequence consisting of gb18030
// second, gb18030 third, and byte, in order.
var buffer = [this.gb18030_second, this.gb18030_third, bite]
// 4. Set gb18030 first, gb18030 second, and gb18030 third to
// 0x00.
this.gb18030_first = 0x00
this.gb18030_second = 0x00
this.gb18030_third = 0x00
// 5. If code point is null, prepend buffer to stream and
// return error.
if (code_point === null) {
stream.prepend(buffer)
return decoderError(this.fatal)
}
// 6. Return a code point whose value is code point.
return code_point
}
// 4. If gb18030 second is not 0x00, run these substeps:
if (this.gb18030_second !== 0x00) {
// 1. If byte is in the range 0x81 to 0xFE, inclusive, set
// gb18030 third to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.gb18030_third = bite
return null
}
// 2. Prepend gb18030 second followed by byte to stream, set
// gb18030 first and gb18030 second to 0x00, and return error.
stream.prepend([this.gb18030_second, bite])
this.gb18030_first = 0x00
this.gb18030_second = 0x00
return decoderError(this.fatal)
}
// 5. If gb18030 first is not 0x00, run these substeps:
if (this.gb18030_first !== 0x00) {
// 1. If byte is in the range 0x30 to 0x39, inclusive, set
// gb18030 second to byte and return continue.
if (inRange(bite, 0x30, 0x39)) {
this.gb18030_second = bite
return null
}
// 2. Let lead be gb18030 first, let pointer be null, and set
// gb18030 first to 0x00.
var lead = this.gb18030_first
var pointer = null
this.gb18030_first = 0x00
// 3. Let offset be 0x40 if byte is less than 0x7F and 0x41
// otherwise.
var offset = bite < 0x7F ? 0x40 : 0x41
// 4. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80
// to 0xFE, inclusive, set pointer to (lead 0x81) × 190 +
// (byte offset).
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0x80, 0xFE))
pointer = (lead - 0x81) * 190 + (bite - offset)
// 5. Let code point be null if pointer is null and the index
// code point for pointer in index gb18030 otherwise.
code_point = pointer === null ? null :
indexCodePointFor(pointer, index('gb18030'))
// 6. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 7. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 8. Return a code point whose value is code point.
return code_point
}
// 6. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 7. If byte is 0x80, return code point U+20AC.
if (bite === 0x80)
return 0x20AC
// 8. If byte is in the range 0x81 to 0xFE, inclusive, set
// gb18030 first to byte and return continue.
if (inRange(bite, 0x81, 0xFE)) {
this.gb18030_first = bite
return null
}
// 9. Return error.
return decoderError(this.fatal)
}
}
// 11.2.2 gb18030 encoder
/**
* @implements {Encoder}
*/
export class GB18030Encoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is U+E5E5, return error with code point.
if (code_point === 0xE5E5)
return encoderError(code_point)
// 4. If the gbk flag is set and code point is U+20AC, return
// byte 0x80.
if (this.gbk_flag && code_point === 0x20AC)
return 0x80
// 5. Let pointer be the index pointer for code point in index
// gb18030.
var pointer = indexPointerFor(code_point, index('gb18030'))
// 6. If pointer is not null, run these substeps:
if (pointer !== null) {
// 1. Let lead be floor(pointer / 190) + 0x81.
var lead = floor(pointer / 190) + 0x81
// 2. Let trail be pointer % 190.
var trail = pointer % 190
// 3. Let offset be 0x40 if trail is less than 0x3F and 0x41 otherwise.
var offset = trail < 0x3F ? 0x40 : 0x41
// 4. Return two bytes whose values are lead and trail + offset.
return [lead, trail + offset]
}
// 7. If gbk flag is set, return error with code point.
if (this.gbk_flag)
return encoderError(code_point)
// 8. Set pointer to the index gb18030 ranges pointer for code
// point.
pointer = indexGB18030RangesPointerFor(code_point)
// 9. Let byte1 be floor(pointer / 10 / 126 / 10).
var byte1 = floor(pointer / 10 / 126 / 10)
// 10. Set pointer to pointer byte1 × 10 × 126 × 10.
pointer = pointer - byte1 * 10 * 126 * 10
// 11. Let byte2 be floor(pointer / 10 / 126).
var byte2 = floor(pointer / 10 / 126)
// 12. Set pointer to pointer byte2 × 10 × 126.
pointer = pointer - byte2 * 10 * 126
// 13. Let byte3 be floor(pointer / 10).
var byte3 = floor(pointer / 10)
// 14. Let byte4 be pointer byte3 × 10.
var byte4 = pointer - byte3 * 10
// 15. Return four bytes whose values are byte1 + 0x81, byte2 +
// 0x30, byte3 + 0x81, byte4 + 0x30.
return [byte1 + 0x81,
byte2 + 0x30,
byte3 + 0x81,
byte4 + 0x30]
}
constructor(options = {}, gbk_flag = false) {
// gb18030's decoder has an associated gbk flag (initially unset).
this.gbk_flag = gbk_flag
}
}
+441
View File
@@ -0,0 +1,441 @@
import { inRange, decoderError, encoderError, isASCIICodePoint,
end_of_stream, finished, floor } from '../utils'
import index, { indexCodePointFor, indexPointerFor } from '../indexes'
// 13.2 iso-2022-jp
// 13.2.1 iso-2022-jp decoder
/**
* @implements {Decoder}
*/
export class ISO2022JPDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
/** @enum */
this.states = {
ASCII: 0,
Roman: 1,
Katakana: 2,
LeadByte: 3,
TrailByte: 4,
EscapeStart: 5,
Escape: 6,
}
// iso-2022-jp's decoder has an associated iso-2022-jp decoder
// state (initially ASCII), iso-2022-jp decoder output state
// (initially ASCII), iso-2022-jp lead (initially 0x00), and
// iso-2022-jp output flag (initially unset).
this.iso2022jp_decoder_state = this.states.ASCII
this.iso2022jp_decoder_output_state = this.states.ASCII,
this.iso2022jp_lead = 0x00
this.iso2022jp_output_flag = false
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// switching on iso-2022-jp decoder state:
switch (this.iso2022jp_decoder_state) {
default:
case this.states.ASCII:
// ASCII
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x00 to 0x7F, excluding 0x0E, 0x0F, and 0x1B
if (inRange(bite, 0x00, 0x7F) && bite !== 0x0E
&& bite !== 0x0F && bite !== 0x1B) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is byte.
this.iso2022jp_output_flag = false
return bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.Roman:
// Roman
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x5C
if (bite === 0x5C) {
// Unset the iso-2022-jp output flag and return code point
// U+00A5.
this.iso2022jp_output_flag = false
return 0x00A5
}
// 0x7E
if (bite === 0x7E) {
// Unset the iso-2022-jp output flag and return code point
// U+203E.
this.iso2022jp_output_flag = false
return 0x203E
}
// 0x00 to 0x7F, excluding 0x0E, 0x0F, 0x1B, 0x5C, and 0x7E
if (inRange(bite, 0x00, 0x7F) && bite !== 0x0E && bite !== 0x0F
&& bite !== 0x1B && bite !== 0x5C && bite !== 0x7E) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is byte.
this.iso2022jp_output_flag = false
return bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.Katakana:
// Katakana
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x21 to 0x5F
if (inRange(bite, 0x21, 0x5F)) {
// Unset the iso-2022-jp output flag and return a code point
// whose value is 0xFF61 0x21 + byte.
this.iso2022jp_output_flag = false
return 0xFF61 - 0x21 + bite
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.LeadByte:
// Lead byte
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return null
}
// 0x21 to 0x7E
if (inRange(bite, 0x21, 0x7E)) {
// Unset the iso-2022-jp output flag, set iso-2022-jp lead
// to byte, iso-2022-jp decoder state to trail byte, and
// return continue.
this.iso2022jp_output_flag = false
this.iso2022jp_lead = bite
this.iso2022jp_decoder_state = this.states.TrailByte
return null
}
// end-of-stream
if (bite === end_of_stream) {
// Return finished.
return finished
}
// Otherwise
// Unset the iso-2022-jp output flag and return error.
this.iso2022jp_output_flag = false
return decoderError(this.fatal)
case this.states.TrailByte:
// Trail byte
// Based on byte:
// 0x1B
if (bite === 0x1B) {
// Set iso-2022-jp decoder state to escape start and return
// continue.
this.iso2022jp_decoder_state = this.states.EscapeStart
return decoderError(this.fatal)
}
// 0x21 to 0x7E
if (inRange(bite, 0x21, 0x7E)) {
// 1. Set the iso-2022-jp decoder state to lead byte.
this.iso2022jp_decoder_state = this.states.LeadByte
// 2. Let pointer be (iso-2022-jp lead 0x21) × 94 + byte 0x21.
const pointer = (this.iso2022jp_lead - 0x21) * 94 + bite - 0x21
// 3. Let code point be the index code point for pointer in
// index jis0208.
const code_point = indexCodePointFor(pointer, index('jis0208'))
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
// end-of-stream
if (bite === end_of_stream) {
// Set the iso-2022-jp decoder state to lead byte, prepend
// byte to stream, and return error.
this.iso2022jp_decoder_state = this.states.LeadByte
stream.prepend(bite)
return decoderError(this.fatal)
}
// Otherwise
// Set iso-2022-jp decoder state to lead byte and return
// error.
this.iso2022jp_decoder_state = this.states.LeadByte
return decoderError(this.fatal)
case this.states.EscapeStart:
// Escape start
// 1. If byte is either 0x24 or 0x28, set iso-2022-jp lead to
// byte, iso-2022-jp decoder state to escape, and return
// continue.
if (bite === 0x24 || bite === 0x28) {
this.iso2022jp_lead = bite
this.iso2022jp_decoder_state = this.states.Escape
return null
}
// 2. Prepend byte to stream.
stream.prepend(bite)
// 3. Unset the iso-2022-jp output flag, set iso-2022-jp
// decoder state to iso-2022-jp decoder output state, and
// return error.
this.iso2022jp_output_flag = false
this.iso2022jp_decoder_state = this.iso2022jp_decoder_output_state
return decoderError(this.fatal)
case this.states.Escape: {
// Escape
// 1. Let lead be iso-2022-jp lead and set iso-2022-jp lead to
// 0x00.
const lead = this.iso2022jp_lead
this.iso2022jp_lead = 0x00
// 2. Let state be null.
let state = null
// 3. If lead is 0x28 and byte is 0x42, set state to ASCII.
if (lead === 0x28 && bite === 0x42)
state = this.states.ASCII
// 4. If lead is 0x28 and byte is 0x4A, set state to Roman.
if (lead === 0x28 && bite === 0x4A)
state = this.states.Roman
// 5. If lead is 0x28 and byte is 0x49, set state to Katakana.
if (lead === 0x28 && bite === 0x49)
state = this.states.Katakana
// 6. If lead is 0x24 and byte is either 0x40 or 0x42, set
// state to lead byte.
if (lead === 0x24 && (bite === 0x40 || bite === 0x42))
state = this.states.LeadByte
// 7. If state is non-null, run these substeps:
if (state !== null) {
// 1. Set iso-2022-jp decoder state and iso-2022-jp decoder
// output state to this.states.
this.iso2022jp_decoder_state = this.iso2022jp_decoder_state = state
// 2. Let output flag be the iso-2022-jp output flag.
const output_flag = this.iso2022jp_output_flag
// 3. Set the iso-2022-jp output flag.
this.iso2022jp_output_flag = true
// 4. Return continue, if output flag is unset, and error
// otherwise.
return !output_flag ? null : decoderError(this.fatal)
}
// 8. Prepend lead and byte to stream.
stream.prepend([lead, bite])
// 9. Unset the iso-2022-jp output flag, set iso-2022-jp
// decoder state to iso-2022-jp decoder output state and
// return error.
this.iso2022jp_output_flag = false
this.iso2022jp_decoder_state = this.iso2022jp_decoder_output_state
return decoderError(this.fatal)
}
}
}
}
// 13.2.2 iso-2022-jp encoder
/**
* @implements {Encoder}
*/
export class ISO2022JPEncoder {
constructor() {
// iso-2022-jp's encoder has an associated iso-2022-jp encoder
// state which is one of ASCII, Roman, and jis0208 (initially
// ASCII).
/** @enum */
this.states = {
ASCII: 0,
Roman: 1,
jis0208: 2,
}
this.iso2022jp_state = this.states.ASCII
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream and iso-2022-jp encoder
// state is not ASCII, prepend code point to stream, set
// iso-2022-jp encoder state to ASCII, and return three bytes
// 0x1B 0x28 0x42.
if (code_point === end_of_stream &&
this.iso2022jp_state !== this.states.ASCII) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.ASCII
return [0x1B, 0x28, 0x42]
}
// 2. If code point is end-of-stream and iso-2022-jp encoder
// state is ASCII, return finished.
if (code_point === end_of_stream && this.iso2022jp_state === this.states.ASCII)
return finished
// 3. If ISO-2022-JP encoder state is ASCII or Roman, and code
// point is U+000E, U+000F, or U+001B, return error with U+FFFD.
if ((this.iso2022jp_state === this.states.ASCII ||
this.iso2022jp_state === this.states.Roman) &&
(code_point === 0x000E || code_point === 0x000F ||
code_point === 0x001B)) {
return encoderError(0xFFFD)
}
// 4. If iso-2022-jp encoder state is ASCII and code point is an
// ASCII code point, return a byte whose value is code point.
if (this.iso2022jp_state === this.states.ASCII &&
isASCIICodePoint(code_point))
return code_point
// 5. If iso-2022-jp encoder state is Roman and code point is an
// ASCII code point, excluding U+005C and U+007E, or is U+00A5
// or U+203E, run these substeps:
if (this.iso2022jp_state === this.states.Roman &&
((isASCIICodePoint(code_point) &&
code_point !== 0x005C && code_point !== 0x007E) ||
(code_point == 0x00A5 || code_point == 0x203E))) {
// 1. If code point is an ASCII code point, return a byte
// whose value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 2. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 3. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
}
// 6. If code point is an ASCII code point, and iso-2022-jp
// encoder state is not ASCII, prepend code point to stream, set
// iso-2022-jp encoder state to ASCII, and return three bytes
// 0x1B 0x28 0x42.
if (isASCIICodePoint(code_point) &&
this.iso2022jp_state !== this.states.ASCII) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.ASCII
return [0x1B, 0x28, 0x42]
}
// 7. If code point is either U+00A5 or U+203E, and iso-2022-jp
// encoder state is not Roman, prepend code point to stream, set
// iso-2022-jp encoder state to Roman, and return three bytes
// 0x1B 0x28 0x4A.
if ((code_point === 0x00A5 || code_point === 0x203E) &&
this.iso2022jp_state !== this.states.Roman) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.Roman
return [0x1B, 0x28, 0x4A]
}
// 8. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 9. Let pointer be the index pointer for code point in index
// jis0208.
const pointer = indexPointerFor(code_point, index('jis0208'))
// 10. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 11. If iso-2022-jp encoder state is not jis0208, prepend code
// point to stream, set iso-2022-jp encoder state to jis0208,
// and return three bytes 0x1B 0x24 0x42.
if (this.iso2022jp_state !== this.states.jis0208) {
stream.prepend(code_point)
this.iso2022jp_state = this.states.jis0208
return [0x1B, 0x24, 0x42]
}
// 12. Let lead be floor(pointer / 94) + 0x21.
const lead = floor(pointer / 94) + 0x21
// 13. Let trail be pointer % 94 + 0x21.
const trail = pointer % 94 + 0x21
// 14. Return two bytes whose values are lead and trail.
return [lead, trail]
}
}
+170
View File
@@ -0,0 +1,170 @@
import { inRange, decoderError, encoderError, floor, isASCIICodePoint, isASCIIByte,
end_of_stream, finished } from '../utils'
import index, { indexCodePointFor, indexShiftJISPointerFor } from '../indexes'
// 13.3 Shift_JIS
// 13.3.1 Shift_JIS decoder
/**
* @constructor
* @implements {Decoder}
* @param {{fatal: boolean}} options
*/
export class ShiftJISDecoder {
constructor(options) {
const { fatal } = options
this.fatal = fatal
// Shift_JIS's decoder has an associated Shift_JIS lead (initially
// 0x00).
this.Shift_JIS_lead = 0x00
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and Shift_JIS lead is not 0x00,
// set Shift_JIS lead to 0x00 and return error.
if (bite === end_of_stream && this.Shift_JIS_lead !== 0x00) {
this.Shift_JIS_lead = 0x00
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and Shift_JIS lead is 0x00,
// return finished.
if (bite === end_of_stream && this.Shift_JIS_lead === 0x00)
return finished
// 3. If Shift_JIS lead is not 0x00, let lead be Shift_JIS lead,
// let pointer be null, set Shift_JIS lead to 0x00, and then run
// these substeps:
if (this.Shift_JIS_lead !== 0x00) {
var lead = this.Shift_JIS_lead
var pointer = null
this.Shift_JIS_lead = 0x00
// 1. Let offset be 0x40, if byte is less than 0x7F, and 0x41
// otherwise.
var offset = (bite < 0x7F) ? 0x40 : 0x41
// 2. Let lead offset be 0x81, if lead is less than 0xA0, and
// 0xC1 otherwise.
var lead_offset = (lead < 0xA0) ? 0x81 : 0xC1
// 3. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80
// to 0xFC, inclusive, set pointer to (lead lead offset) ×
// 188 + byte offset.
if (inRange(bite, 0x40, 0x7E) || inRange(bite, 0x80, 0xFC))
pointer = (lead - lead_offset) * 188 + bite - offset
// 4. If pointer is in the range 8836 to 10715, inclusive,
// return a code point whose value is 0xE000 8836 + pointer.
if (inRange(pointer, 8836, 10715))
return 0xE000 - 8836 + pointer
// 5. Let code point be null, if pointer is null, and the
// index code point for pointer in index jis0208 otherwise.
var code_point = (pointer === null) ? null :
indexCodePointFor(pointer, index('jis0208'))
// 6. If code point is null and byte is an ASCII byte, prepend
// byte to stream.
if (code_point === null && isASCIIByte(bite))
stream.prepend(bite)
// 7. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 8. Return a code point whose value is code point.
return code_point
}
// 4. If byte is an ASCII byte or 0x80, return a code point
// whose value is byte.
if (isASCIIByte(bite) || bite === 0x80)
return bite
// 5. If byte is in the range 0xA1 to 0xDF, inclusive, return a
// code point whose value is 0xFF61 0xA1 + byte.
if (inRange(bite, 0xA1, 0xDF))
return 0xFF61 - 0xA1 + bite
// 6. If byte is in the range 0x81 to 0x9F, inclusive, or 0xE0
// to 0xFC, inclusive, set Shift_JIS lead to byte and return
// continue.
if (inRange(bite, 0x81, 0x9F) || inRange(bite, 0xE0, 0xFC)) {
this.Shift_JIS_lead = bite
return null
}
// 7. Return error.
return decoderError(this.fatal)
}
}
// 13.3.2 Shift_JIS encoder
/**
* @constructor
* @implements {Encoder}
* @param {{fatal: boolean}} options
*/
export class ShiftJISEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point or U+0080, return a
// byte whose value is code point.
if (isASCIICodePoint(code_point) || code_point === 0x0080)
return code_point
// 3. If code point is U+00A5, return byte 0x5C.
if (code_point === 0x00A5)
return 0x5C
// 4. If code point is U+203E, return byte 0x7E.
if (code_point === 0x203E)
return 0x7E
// 5. If code point is in the range U+FF61 to U+FF9F, inclusive,
// return a byte whose value is code point 0xFF61 + 0xA1.
if (inRange(code_point, 0xFF61, 0xFF9F))
return code_point - 0xFF61 + 0xA1
// 6. If code point is U+2212, set it to U+FF0D.
if (code_point === 0x2212)
code_point = 0xFF0D
// 7. Let pointer be the index Shift_JIS pointer for code point.
var pointer = indexShiftJISPointerFor(code_point)
// 8. If pointer is null, return error with code point.
if (pointer === null)
return encoderError(code_point)
// 9. Let lead be floor(pointer / 188).
var lead = floor(pointer / 188)
// 10. Let lead offset be 0x81, if lead is less than 0x1F, and
// 0xC1 otherwise.
var lead_offset = (lead < 0x1F) ? 0x81 : 0xC1
// 11. Let trail be pointer % 188.
var trail = pointer % 188
// 12. Let offset be 0x40, if trail is less than 0x3F, and 0x41
// otherwise.
var offset = (trail < 0x3F) ? 0x40 : 0x41
// 13. Return two bytes whose values are lead + lead offset and
// trail + offset.
return [lead + lead_offset, trail + offset]
}
}
+86
View File
@@ -0,0 +1,86 @@
import { end_of_stream, finished, isASCIIByte, decoderError, encoderError, isASCIICodePoint } from '../utils'
import { indexPointerFor } from '../indexes'
//
// 10. Legacy single-byte encodings
//
// 10.1 single-byte decoder
/**
* @implements {Decoder}
*/
export class SingleByteDecoder {
/**
* @param {!Array.<number>} index The encoding index.
* @param {{fatal: boolean}} options
*/
constructor(index, options) {
const { fatal } = options
this.fatal = fatal
this.index = index
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 2. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 3. Let code point be the index code point for byte 0x80 in
// index single-byte.
var code_point = this.index[bite - 0x80]
// 4. If code point is null, return error.
if (code_point === null)
return decoderError(this.fatal)
// 5. Return a code point whose value is code point.
return code_point
}
}
// 10.2 single-byte encoder
/**
* @implements {Encoder}
*/
export class SingleByteEncoder {
/**
* @param {!Array.<?number>} index The encoding index.
*/
constructor(index) {
this.index = index
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Let pointer be the index pointer for code point in index
// single-byte.
const pointer = indexPointerFor(code_point, this.index)
// 4. If pointer is null, return error with code point.
if (pointer === null)
encoderError(code_point)
// 5. Return a byte whose value is pointer + 0x80.
return pointer + 0x80
}
}
+139
View File
@@ -0,0 +1,139 @@
import { inRange, decoderError, end_of_stream, finished, convertCodeUnitToBytes } from '../utils'
// 15.2.1 shared utf-16 decoder
/**
* @implements {Decoder}
*/
export class UTF16Decoder {
/**
* @param {boolean} utf16_be True if big-endian, false if little-endian.
* @param {{fatal: boolean}} options
*/
constructor(utf16_be, options) {
const { fatal } = options
this.utf16_be = utf16_be
this.fatal = fatal
this.utf16_lead_byte = null
this.utf16_lead_surrogate = null
}
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream and either utf-16 lead byte or
// utf-16 lead surrogate is not null, set utf-16 lead byte and
// utf-16 lead surrogate to null, and return error.
if (bite === end_of_stream && (this.utf16_lead_byte !== null ||
this.utf16_lead_surrogate !== null)) {
return decoderError(this.fatal)
}
// 2. If byte is end-of-stream and utf-16 lead byte and utf-16
// lead surrogate are null, return finished.
if (bite === end_of_stream && this.utf16_lead_byte === null &&
this.utf16_lead_surrogate === null) {
return finished
}
// 3. If utf-16 lead byte is null, set utf-16 lead byte to byte
// and return continue.
if (this.utf16_lead_byte === null) {
this.utf16_lead_byte = bite
return null
}
// 4. Let code unit be the result of:
let code_unit
if (this.utf16_be) {
// utf-16be decoder flag is set
// (utf-16 lead byte << 8) + byte.
code_unit = (this.utf16_lead_byte << 8) + bite
} else {
// utf-16be decoder flag is unset
// (byte << 8) + utf-16 lead byte.
code_unit = (bite << 8) + this.utf16_lead_byte
}
// Then set utf-16 lead byte to null.
this.utf16_lead_byte = null
// 5. If utf-16 lead surrogate is not null, let lead surrogate
// be utf-16 lead surrogate, set utf-16 lead surrogate to null,
// and then run these substeps:
if (this.utf16_lead_surrogate !== null) {
const lead_surrogate = this.utf16_lead_surrogate
this.utf16_lead_surrogate = null
// 1. If code unit is in the range U+DC00 to U+DFFF,
// inclusive, return a code point whose value is 0x10000 +
// ((lead surrogate 0xD800) << 10) + (code unit 0xDC00).
if (inRange(code_unit, 0xDC00, 0xDFFF)) {
return 0x10000 + (lead_surrogate - 0xD800) * 0x400 +
(code_unit - 0xDC00)
}
// 2. Prepend the sequence resulting of converting code unit
// to bytes using utf-16be decoder flag to stream and return
// error.
stream.prepend(convertCodeUnitToBytes(code_unit, this.utf16_be))
return decoderError(this.fatal)
}
// 6. If code unit is in the range U+D800 to U+DBFF, inclusive,
// set utf-16 lead surrogate to code unit and return continue.
if (inRange(code_unit, 0xD800, 0xDBFF)) {
this.utf16_lead_surrogate = code_unit
return null
}
// 7. If code unit is in the range U+DC00 to U+DFFF, inclusive,
// return error.
if (inRange(code_unit, 0xDC00, 0xDFFF))
return decoderError(this.fatal)
// 8. Return code point code unit.
return code_unit
}
}
// 15.2.2 shared utf-16 encoder
/**
* @implements {Encoder}
*/
export class UTF16Encoder {
/**
* @param {boolean} [utf16_be] True if big-endian, false if little-endian.
*/
constructor(utf16_be = false) {
this.utf16_be = utf16_be
}
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is in the range U+0000 to U+FFFF, inclusive,
// return the sequence resulting of converting code point to
// bytes using utf-16be encoder flag.
if (inRange(code_point, 0x0000, 0xFFFF))
return convertCodeUnitToBytes(code_point, this.utf16_be)
// 3. Let lead be ((code point 0x10000) >> 10) + 0xD800,
// converted to bytes using utf-16be encoder flag.
const lead = convertCodeUnitToBytes(
((code_point - 0x10000) >> 10) + 0xD800, this.utf16_be)
// 4. Let trail be ((code point 0x10000) & 0x3FF) + 0xDC00,
// converted to bytes using utf-16be encoder flag.
const trail = convertCodeUnitToBytes(
((code_point - 0x10000) & 0x3FF) + 0xDC00, this.utf16_be)
// 5. Return a byte sequence of lead followed by trail.
return lead.concat(trail)
}
}
+208
View File
@@ -0,0 +1,208 @@
import { inRange, decoderError, isASCIICodePoint,
end_of_stream, finished } from '../utils'
/**
* @implements {Decoder}
*/
export class UTF8Decoder {
/**
* @param {{fatal: boolean}} options
*/
constructor(options) {
const { fatal } = options
// utf-8's decoder's has an associated utf-8 code point, utf-8
// bytes seen, and utf-8 bytes needed (all initially 0), a utf-8
// lower boundary (initially 0x80), and a utf-8 upper boundary
// (initially 0xBF).
let /** @type {number} */ utf8_code_point = 0,
/** @type {number} */ utf8_bytes_seen = 0,
/** @type {number} */ utf8_bytes_needed = 0,
/** @type {number} */ utf8_lower_boundary = 0x80,
/** @type {number} */ utf8_upper_boundary = 0xBF
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
* @return {?(number|!Array.<number>)} The next code point(s)
* decoded, or null if not enough data exists in the input
* stream to decode a complete code point.
*/
this.handler = function(stream, bite) {
// 1. If byte is end-of-stream and utf-8 bytes needed is not 0,
// set utf-8 bytes needed to 0 and return error.
if (bite === end_of_stream && utf8_bytes_needed !== 0) {
utf8_bytes_needed = 0
return decoderError(fatal)
}
// 2. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 3. If utf-8 bytes needed is 0, based on byte:
if (utf8_bytes_needed === 0) {
// 0x00 to 0x7F
if (inRange(bite, 0x00, 0x7F)) {
// Return a code point whose value is byte.
return bite
}
// 0xC2 to 0xDF
else if (inRange(bite, 0xC2, 0xDF)) {
// 1. Set utf-8 bytes needed to 1.
utf8_bytes_needed = 1
// 2. Set UTF-8 code point to byte & 0x1F.
utf8_code_point = bite & 0x1F
}
// 0xE0 to 0xEF
else if (inRange(bite, 0xE0, 0xEF)) {
// 1. If byte is 0xE0, set utf-8 lower boundary to 0xA0.
if (bite === 0xE0)
utf8_lower_boundary = 0xA0
// 2. If byte is 0xED, set utf-8 upper boundary to 0x9F.
if (bite === 0xED)
utf8_upper_boundary = 0x9F
// 3. Set utf-8 bytes needed to 2.
utf8_bytes_needed = 2
// 4. Set UTF-8 code point to byte & 0xF.
utf8_code_point = bite & 0xF
}
// 0xF0 to 0xF4
else if (inRange(bite, 0xF0, 0xF4)) {
// 1. If byte is 0xF0, set utf-8 lower boundary to 0x90.
if (bite === 0xF0)
utf8_lower_boundary = 0x90
// 2. If byte is 0xF4, set utf-8 upper boundary to 0x8F.
if (bite === 0xF4)
utf8_upper_boundary = 0x8F
// 3. Set utf-8 bytes needed to 3.
utf8_bytes_needed = 3
// 4. Set UTF-8 code point to byte & 0x7.
utf8_code_point = bite & 0x7
}
// Otherwise
else {
// Return error.
return decoderError(fatal)
}
// Return continue.
return null
}
// 4. If byte is not in the range utf-8 lower boundary to utf-8
// upper boundary, inclusive, run these substeps:
if (!inRange(bite, utf8_lower_boundary, utf8_upper_boundary)) {
// 1. Set utf-8 code point, utf-8 bytes needed, and utf-8
// bytes seen to 0, set utf-8 lower boundary to 0x80, and set
// utf-8 upper boundary to 0xBF.
utf8_code_point = utf8_bytes_needed = utf8_bytes_seen = 0
utf8_lower_boundary = 0x80
utf8_upper_boundary = 0xBF
// 2. Prepend byte to stream.
stream.prepend(bite)
// 3. Return error.
return decoderError(fatal)
}
// 5. Set utf-8 lower boundary to 0x80 and utf-8 upper boundary
// to 0xBF.
utf8_lower_boundary = 0x80
utf8_upper_boundary = 0xBF
// 6. Set UTF-8 code point to (UTF-8 code point << 6) | (byte &
// 0x3F)
utf8_code_point = (utf8_code_point << 6) | (bite & 0x3F)
// 7. Increase utf-8 bytes seen by one.
utf8_bytes_seen += 1
// 8. If utf-8 bytes seen is not equal to utf-8 bytes needed,
// continue.
if (utf8_bytes_seen !== utf8_bytes_needed)
return null
// 9. Let code point be utf-8 code point.
var code_point = utf8_code_point
// 10. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes
// seen to 0.
utf8_code_point = utf8_bytes_needed = utf8_bytes_seen = 0
// 11. Return a code point whose value is code point.
return code_point
}
}
}
// 9.1.2 utf-8 encoder
/**
* @implements {Encoder}
*/
export class UTF8Encoder {
constructor() {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit.
*/
this.handler = function(stream, code_point) {
// 1. If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. Set count and offset based on the range code point is in:
var count, offset
// U+0080 to U+07FF, inclusive:
if (inRange(code_point, 0x0080, 0x07FF)) {
// 1 and 0xC0
count = 1
offset = 0xC0
}
// U+0800 to U+FFFF, inclusive:
else if (inRange(code_point, 0x0800, 0xFFFF)) {
// 2 and 0xE0
count = 2
offset = 0xE0
}
// U+10000 to U+10FFFF, inclusive:
else if (inRange(code_point, 0x10000, 0x10FFFF)) {
// 3 and 0xF0
count = 3
offset = 0xF0
}
// 4. Let bytes be a byte sequence whose first byte is (code
// point >> (6 × count)) + offset.
var bytes = [(code_point >> (6 * count)) + offset]
// 5. Run these substeps while count is greater than 0:
while (count > 0) {
// 1. Set temp to code point >> (6 × (count 1)).
var temp = code_point >> (6 * (count - 1))
// 2. Append to bytes 0x80 | (temp & 0x3F).
bytes.push(0x80 | (temp & 0x3F))
// 3. Decrease count by one.
count -= 1
}
// 6. Return bytes bytes, in order.
return bytes
}
}
}
+56
View File
@@ -0,0 +1,56 @@
import { inRange, encoderError, end_of_stream, finished, isASCIIByte, isASCIICodePoint } from '../utils'
// 15.5 x-user-defined
// 15.5.1 x-user-defined decoder
/**
* @implements {Decoder}
*/
export class XUserDefinedDecoder {
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
*/
handler(stream, bite) {
// 1. If byte is end-of-stream, return finished.
if (bite === end_of_stream)
return finished
// 2. If byte is an ASCII byte, return a code point whose value
// is byte.
if (isASCIIByte(bite))
return bite
// 3. Return a code point whose value is 0xF780 + byte 0x80.
return 0xF780 + bite - 0x80
}
}
// 15.5.2 x-user-defined encoder
/**
* @implements {Encoder}
*/
export class XUserDefinedEncoder {
/**
* @param {Stream} stream Input stream.
* @param {number} code_point Next code point read from the stream.
*/
handler(stream, code_point) {
// 1.If code point is end-of-stream, return finished.
if (code_point === end_of_stream)
return finished
// 2. If code point is an ASCII code point, return a byte whose
// value is code point.
if (isASCIICodePoint(code_point))
return code_point
// 3. If code point is in the range U+F780 to U+F7FF, inclusive,
// return a byte whose value is code point 0xF780 + 0x80.
if (inRange(code_point, 0xF780, 0xF7FF))
return code_point - 0xF780 + 0x80
// 4. Return error with code point.
return encoderError(code_point)
}
}
+11
View File
@@ -0,0 +1,11 @@
import TextEncoder from './lib/TextEncoder'
import TextDecoder from './lib/TextDecoder'
import EncodingIndexes from './encoding-indexes'
import { getEncoding } from './lib'
//
// Implementation of Encoding specification
// https://encoding.spec.whatwg.org/
//
export { TextEncoder, TextDecoder, EncodingIndexes, getEncoding }
+153
View File
@@ -0,0 +1,153 @@
import { inRange } from './utils'
import Indexes from './encoding-indexes'
//
// 6. Indexes
//
/**
* @param {number} pointer The |pointer| to search for.
* @param {(!Array.<?number>|undefined)} index The |index| to search within.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in |index|.
*/
export function indexCodePointFor(pointer, i) {
if (!i) return null
return i[pointer] || null
}
/**
* @param {number} code_point The |code point| to search for.
* @param {!Array.<?number>} i The |index| to search within.
* @return {?number} The first pointer corresponding to |code point| in
* |index|, or null if |code point| is not in |index|.
*/
export function indexPointerFor(code_point, i) {
var pointer = i.indexOf(code_point)
return pointer === -1 ? null : pointer
}
/**
* @param {string} name Name of the index.
*/
export default function index(name) {
return Indexes[name]
}
/**
* @param {number} pointer The |pointer| to search for in the gb18030 index.
* @return The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the gb18030 index.
*/
export function indexGB18030RangesCodePointFor(pointer) {
// 1. If pointer is greater than 39419 and less than 189000, or
// pointer is greater than 1237575, return null.
if ((pointer > 39419 && pointer < 189000) || (pointer > 1237575))
return null
// 2. If pointer is 7457, return code point U+E7C7.
if (pointer === 7457) return 0xE7C7
// 3. Let offset be the last pointer in index gb18030 ranges that
// is equal to or less than pointer and let code point offset be
// its corresponding code point.
var offset = 0
var code_point_offset = 0
var idx = index('gb18030-ranges')
var i
for (i = 0; i < idx.length; ++i) {
/** @type {!Array.<number>} */
var entry = idx[i]
if (entry[0] <= pointer) {
offset = entry[0]
code_point_offset = entry[1]
} else {
break
}
}
// 4. Return a code point whose value is code point offset +
// pointer offset.
return code_point_offset + pointer - offset
}
/**
* @param {number} code_point The |code point| to locate in the gb18030 index.
* @return {number} The first pointer corresponding to |code point| in the
* gb18030 index.
*/
export function indexGB18030RangesPointerFor(code_point) {
// 1. If code point is U+E7C7, return pointer 7457.
if (code_point === 0xE7C7) return 7457
// 2. Let offset be the last code point in index gb18030 ranges
// that is equal to or less than code point and let pointer offset
// be its corresponding pointer.
var offset = 0
var pointer_offset = 0
var idx = index('gb18030-ranges')
var i
for (i = 0; i < idx.length; ++i) {
/** @type {!Array.<number>} */
var entry = idx[i]
if (entry[1] <= code_point) {
offset = entry[1]
pointer_offset = entry[0]
} else {
break
}
}
// 3. Return a pointer whose value is pointer offset + code point
// offset.
return pointer_offset + code_point - offset
}
/**
* @param {number} code_point The |code_point| to search for in the Shift_JIS
* index.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the Shift_JIS index.
*/
export function indexShiftJISPointerFor(code_point) {
// 1. Let index be index jis0208 excluding all entries whose
// pointer is in the range 8272 to 8835, inclusive.
shift_jis_index = shift_jis_index ||
index('jis0208').map((cp, pointer) => {
return inRange(pointer, 8272, 8835) ? null : cp
})
const index_ = shift_jis_index
// 2. Return the index pointer for code point in index.
return index_.indexOf(code_point)
}
var shift_jis_index
/**
* @param {number} code_point The |code_point| to search for in the big5
* index.
* @return {?number} The code point corresponding to |pointer| in |index|,
* or null if |code point| is not in the big5 index.
*/
export function indexBig5PointerFor(code_point) {
// 1. Let index be index Big5 excluding all entries whose pointer
big5_index_no_hkscs = big5_index_no_hkscs ||
index('big5').map((cp, pointer) => {
return (pointer < (0xA1 - 0x81) * 157) ? null : cp
})
var index_ = big5_index_no_hkscs
// 2. If code point is U+2550, U+255E, U+2561, U+256A, U+5341, or
// U+5345, return the last pointer corresponding to code point in
// index.
if (code_point === 0x2550 || code_point === 0x255E ||
code_point === 0x2561 || code_point === 0x256A ||
code_point === 0x5341 || code_point === 0x5345) {
return index_.lastIndexOf(code_point)
}
// 3. Return the index pointer for code point in index.
return indexPointerFor(code_point, index_)
}
var big5_index_no_hkscs
+189
View File
@@ -0,0 +1,189 @@
import Stream, { DEFAULT_ENCODING, getEncoding } from './'
import { end_of_stream, finished, codePointsToString } from '../utils'
import { decoders } from '../table'
// 8.1 Interface TextDecoder
export default class TextDecoder {
/**
* @param {string=} label The label of the encoding; defaults to 'utf-8'.
* @param {Object=} options
*/
constructor(label = DEFAULT_ENCODING, options = {}) {
// A TextDecoder object has an associated encoding, decoder,
// stream, ignore BOM flag (initially unset), BOM seen flag
// (initially unset), error mode (initially replacement), and do
// not flush flag (initially unset).
/** @private */
this._encoding = null
/** @private @type {?Decoder} */
this._decoder = null
/** @private @type {boolean} */
this._ignoreBOM = false
/** @private @type {boolean} */
this._BOMseen = false
/** @private @type {string} */
this._error_mode = 'replacement'
/** @private @type {boolean} */
this._do_not_flush = false
// 1. Let encoding be the result of getting an encoding from
// label.
const encoding = getEncoding(label)
// 2. If encoding is failure or replacement, throw a RangeError.
if (encoding === null || encoding.name == 'replacement')
throw RangeError('Unknown encoding: ' + label)
if (!decoders[encoding.name]) {
throw Error('Decoder not present.' +
' Did you forget to include encoding-indexes.js first?')
}
// 4. Set dec's encoding to encoding.
this._encoding = encoding
// 5. If options's fatal member is true, set dec's error mode to
// fatal.
if (options['fatal'])
this._error_mode = 'fatal'
// 6. If options's ignoreBOM member is true, set dec's ignore BOM
// flag.
if (options['ignoreBOM'])
this._ignoreBOM = true
}
get encoding() {
return this._encoding.name.toLowerCase()
}
get fatal() {
return this._error_mode === 'fatal'
}
get ignoreBOM() {
return this._ignoreBOM
}
/**
* @param {BufferSource=} input The buffer of bytes to decode.
* @param {Object=} options
* @return The decoded string.
*/
decode(input, options = {}) {
let bytes
if (typeof input === 'object' && input instanceof ArrayBuffer) {
bytes = new Uint8Array(input)
} else if (typeof input === 'object' && 'buffer' in input &&
input.buffer instanceof ArrayBuffer) {
bytes = new Uint8Array(input.buffer,
input.byteOffset,
input.byteLength)
} else {
bytes = new Uint8Array(0)
}
// 1. If the do not flush flag is unset, set decoder to a new
// encoding's decoder, set stream to a new stream, and unset the
// BOM seen flag.
if (!this._do_not_flush) {
this._decoder = decoders[this._encoding.name]({
fatal: this._error_mode === 'fatal' })
this._BOMseen = false
}
// 2. If options's stream is true, set the do not flush flag, and
// unset the do not flush flag otherwise.
this._do_not_flush = Boolean(options['stream'])
// 3. If input is given, push a copy of input to stream.
// TODO: Align with spec algorithm - maintain stream on instance.
const input_stream = new Stream(bytes)
// 4. Let output be a new stream.
const output = []
/** @type {?(number|!Array.<number>)} */
let result
// 5. While true:
while (true) {
// 1. Let token be the result of reading from stream.
const token = input_stream.read()
// 2. If token is end-of-stream and the do not flush flag is
// set, return output, serialized.
// TODO: Align with spec algorithm.
if (token === end_of_stream)
break
// 3. Otherwise, run these subsubsteps:
// 1. Let result be the result of processing token for decoder,
// stream, output, and error mode.
result = this._decoder.handler(input_stream, token)
// 2. If result is finished, return output, serialized.
if (result === finished)
break
if (result !== null) {
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
// 3. Otherwise, if result is error, throw a TypeError.
// (Thrown in handler)
// 4. Otherwise, do nothing.
}
// TODO: Align with spec algorithm.
if (!this._do_not_flush) {
do {
result = this._decoder.handler(input_stream, input_stream.read())
if (result === finished)
break
if (result === null)
continue
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
} while (!input_stream.endOfStream())
this._decoder = null
}
return this.serializeStream(output)
}
// A TextDecoder object also has an associated serialize stream
// algorithm...
/**
* @param {!Array.<number>} stream
*/
serializeStream(stream) {
// 1. Let token be the result of reading from stream.
// (Done in-place on array, rather than as a stream)
// 2. If encoding is UTF-8, UTF-16BE, or UTF-16LE, and ignore
// BOM flag and BOM seen flag are unset, run these subsubsteps:
if (['UTF-8', 'UTF-16LE', 'UTF-16BE'].includes(this._encoding.name) &&
!this._ignoreBOM && !this._BOMseen) {
if (stream.length > 0 && stream[0] === 0xFEFF) {
// 1. If token is U+FEFF, set BOM seen flag.
this._BOMseen = true
stream.shift()
} else if (stream.length > 0) {
// 2. Otherwise, if token is not end-of-stream, set BOM seen
// flag and append token to stream.
this._BOMseen = true
} else {
// 3. Otherwise, if token is not end-of-stream, append token
// to output.
// (no-op)
}
}
// 4. Otherwise, return output.
return codePointsToString(stream)
}
}
+106
View File
@@ -0,0 +1,106 @@
import Stream, { DEFAULT_ENCODING, getEncoding } from './'
import { end_of_stream, finished, stringToCodePoints } from '../utils'
import { encoders } from '../table'
// 8.2 Interface TextEncoder
export default class TextEncoder {
/**
* @param {string=} label The label of the encoding. NONSTANDARD.
* @param {Object=} [options] NONSTANDARD.
*/
constructor(label, options = {}) {
// A TextEncoder object has an associated encoding and encoder.
/** @private */
this._encoding = null
/** @private @type {?Encoder} */
this._encoder = null
// Non-standard
/** @private @type {boolean} */
this._do_not_flush = false
/** @private @type {string} */
this._fatal = options['fatal'] ? 'fatal' : 'replacement'
// 2. Set enc's encoding to UTF-8's encoder.
if (options['NONSTANDARD_allowLegacyEncoding']) {
// NONSTANDARD behavior.
label = label !== undefined ? String(label) : DEFAULT_ENCODING
var encoding = getEncoding(label)
if (encoding === null || encoding.name === 'replacement')
throw RangeError('Unknown encoding: ' + label)
if (!encoders[encoding.name]) {
throw Error('Encoder not present.' +
' Did you forget to include encoding-indexes.js first?')
}
this._encoding = encoding
} else {
// Standard behavior.
this._encoding = getEncoding('utf-8')
if (label !== undefined && 'console' in global) {
console.warn('TextEncoder constructor called with encoding label, '
+ 'which is ignored.')
}
}
}
get encoding() {
return this._encoding.name.toLowerCase()
}
/**
* @param {string=} opt_string The string to encode.
* @param {Object=} options
*/
encode(opt_string = '', options = {}) {
// NOTE: This option is nonstandard. None of the encodings
// permitted for encoding (i.e. UTF-8, UTF-16) are stateful when
// the input is a USVString so streaming is not necessary.
if (!this._do_not_flush)
this._encoder = encoders[this._encoding.name]({
fatal: this._fatal === 'fatal' })
this._do_not_flush = Boolean(options['stream'])
// 1. Convert input to a stream.
const input = new Stream(stringToCodePoints(opt_string))
// 2. Let output be a new stream
const output = []
/** @type {?(number|!Array.<number>)} */
var result
// 3. While true, run these substeps:
while (true) {
// 1. Let token be the result of reading from input.
var token = input.read()
if (token === end_of_stream)
break
// 2. Let result be the result of processing token for encoder,
// input, output.
result = this._encoder.handler(input, token)
if (result === finished)
break
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
// TODO: Align with spec algorithm.
if (!this._do_not_flush) {
while (true) {
result = this._encoder.handler(input, input.read())
if (result === finished)
break
if (Array.isArray(result))
output.push.apply(output, /**@type {!Array.<number>}*/(result))
else
output.push(result)
}
this._encoder = null
}
// 3. If result is finished, convert output into a byte sequence,
// and then return a Uint8Array object wrapping an ArrayBuffer
// containing output.
return new Uint8Array(output)
}
}
+118
View File
@@ -0,0 +1,118 @@
import { end_of_stream } from '../utils'
import { label_to_encoding } from '../table'
export default class Stream {
/**
* A stream represents an ordered sequence of tokens.
* @param {!(Array.<number>|Uint8Array)} tokens Array of tokens that provide
* the stream.
*/
constructor(tokens) {
this.tokens = [...tokens]
// Reversed as push/pop is more efficient than shift/unshift.
this.tokens.reverse()
}
/**
* @returns True if end-of-stream has been hit.
*/
endOfStream() {
return !this.tokens.length
}
/**
* When a token is read from a stream, the first token in the
* stream must be returned and subsequently removed, and
* end-of-stream must be returned otherwise.
*
* @return Get the next token from the stream, or end_of_stream.
*/
read() {
if (!this.tokens.length)
return end_of_stream
return this.tokens.pop()
}
/**
* When one or more tokens are prepended to a stream, those tokens
* must be inserted, in given order, before the first token in the
* stream.
*
* @param {(number|!Array.<number>)} token The token(s) to prepend to the
* stream.
*/
prepend(token) {
if (Array.isArray(token)) {
var tokens = /**@type {!Array.<number>}*/(token)
while (tokens.length)
this.tokens.push(tokens.pop())
} else {
this.tokens.push(token)
}
}
/**
* When one or more tokens are pushed to a stream, those tokens
* must be inserted, in given order, after the last token in the
* stream.
*
* @param {(number|!Array.<number>)} token The tokens(s) to push to the
* stream.
*/
push(token) {
if (Array.isArray(token)) {
const tokens = /**@type {!Array.<number>}*/(token)
while (tokens.length)
this.tokens.unshift(tokens.shift())
} else {
this.tokens.unshift(token)
}
}
}
export const DEFAULT_ENCODING = 'utf-8'
/**
* Returns the encoding for the label.
* @param {string} label The encoding label.
*/
export function getEncoding(label) {
// 1. Remove any leading and trailing ASCII whitespace from label.
label = String(label).trim().toLowerCase()
// 2. If label is an ASCII case-insensitive match for any of the
// labels listed in the table below, return the corresponding
// encoding, and failure otherwise.
if (Object.prototype.hasOwnProperty.call(label_to_encoding, label)) {
return label_to_encoding[label]
}
return null
}
//
// 5. Encodings
//
// 5.1 Encoders and decoders
// /** @interface */
// function Decoder() {}
// Decoder.prototype = {
// /**
// * @param {Stream} stream The stream of bytes being decoded.
// * @param {number} bite The next byte read from the stream.
// * @return {?(number|!Array.<number>)} The next code point(s)
// * decoded, or null if not enough data exists in the input
// * stream to decode a complete code point, or |finished|.
// */
// handler: function(stream, bite) {},
// }
// /** @interface */
// function Encoder() {}
// Encoder.prototype = {
// /**
// * @param {Stream} stream The stream of code points being encoded.
// * @param {number} code_point Next code point read from the stream.
// * @return {(number|!Array.<number>)} Byte(s) to emit, or |finished|.
// */
// handler: function(stream, code_point) {},
// }
+119
View File
@@ -0,0 +1,119 @@
import Encodings from './encodings'
import { UTF8Decoder, UTF8Encoder } from './implementations/utf8'
import { UTF16Decoder, UTF16Encoder } from './implementations/utf16'
import { GB18030Decoder, GB18030Encoder } from './implementations/gb18030'
import { Big5Decoder, Big5Encoder } from './implementations/big5'
import { EUCJPDecoder, EUCJPEncoder } from './implementations/euc-jp'
import { EUCKRDecoder, EUCKREncoder } from './implementations/euc-kr'
import { ISO2022JPDecoder, ISO2022JPEncoder } from './implementations/iso-2022-jp'
import { XUserDefinedDecoder, XUserDefinedEncoder } from './implementations/x-user-defined'
import { ShiftJISDecoder, ShiftJISEncoder } from './implementations/shift-jis'
import { SingleByteDecoder, SingleByteEncoder } from './implementations/single-byte'
import index from './indexes';
// 5.2 Names and labels
// TODO: Define @typedef for Encoding: {name:string,labels:Array.<string>}
// https://github.com/google/closure-compiler/issues/247
// Label to encoding registry.
/** @type {Object.<string,{name:string,labels:Array.<string>}>} */
export const label_to_encoding = {}
Encodings.forEach(({ encodings }) => {
encodings.forEach((encoding) => {
encoding.labels.forEach((label) => {
label_to_encoding[label] = encoding
})
})
})
// Registry of of encoder/decoder factories, by encoding name.
export const encoders = {
'UTF-8'() { // 9.1 utf-8
return new UTF8Encoder()
},
'GBK'(options) { // 11.1.2 gbk encoder;
// gbk's encoder is gb18030's encoder with its gbk flag set.
return new GB18030Encoder(options, true)
},
'gb18030'() {
return new GB18030Encoder()
},
'Big5'() {
return new Big5Encoder()
},
'EUC-JP'() {
return new EUCJPEncoder()
},
'EUC-KR'() {
return new EUCKREncoder()
},
'ISO-2022-JP'() {
return new ISO2022JPEncoder()
},
'UTF-16BE'() { // 15.3 utf-16be
return new UTF16Encoder(true)
},
'UTF-16LE'() { // 15.4 utf-16le
return new UTF16Encoder()
},
'x-user-defined'() {
return new XUserDefinedEncoder()
},
'Shift_JIS'() {
return new ShiftJISEncoder()
},
}
/** @type {Object.<string, function({fatal:boolean}): Decoder>} */
export const decoders = {
'UTF-8'(options) { // 9.1.1 utf-8 decoder
return new UTF8Decoder(options)
},
'GBK'(options) { // 11.1.1 gbk decoder; gbk's decoder is gb18030's decoder.
return new GB18030Decoder(options)
},
'gb18030'(options) {
return new GB18030Decoder(options)
},
'Big5'(options) {
return new Big5Decoder(options)
},
'EUC-JP'(options) {
return new EUCJPDecoder(options)
},
'EUC-KR'(options) {
return new EUCKRDecoder(options)
},
'ISO-2022-JP'(options) {
return new ISO2022JPDecoder(options)
},
'UTF-16BE'(options) { // 15.3.1 utf-16be decoder
return new UTF16Decoder(true, options)
},
'UTF-16LE'(options) { // 15.4.1 utf-16le decoder
return new UTF16Decoder(false, options)
},
'x-user-defined'() {
return new XUserDefinedDecoder()
},
'Shift_JIS'(options) {
return new ShiftJISDecoder(options)
},
}
Encodings.forEach(({ heading, encodings }) => {
if (heading != 'Legacy single-byte encodings')
return
encodings.forEach((encoding) => {
const name = encoding.name
const idx = index(name.toLowerCase())
decoders[name] = (options) => {
return new SingleByteDecoder(idx, options)
}
encoders[name] = (options) => {
return new SingleByteEncoder(idx, options)
}
})
})
+180
View File
@@ -0,0 +1,180 @@
//
// Utilities
//
/**
* @param {number} a The number to test.
* @param {number} min The minimum value in the range, inclusive.
* @param {number} max The maximum value in the range, inclusive.
* @return {boolean} True if a >= min and a <= max.
*/
export function inRange(a, min, max) {
return min <= a && a <= max
}
export const floor = Math.floor
/**
* @param {string} string Input string of UTF-16 code units.
* @return {!Array.<number>} Code points.
*/
export function stringToCodePoints(string) {
// https://heycam.github.io/webidl/#dfn-obtain-unicode
// 1. Let S be the DOMString value.
var s = String(string)
// 2. Let n be the length of S.
var n = s.length
// 3. Initialize i to 0.
var i = 0
// 4. Initialize U to be an empty sequence of Unicode characters.
var u = []
// 5. While i < n:
while (i < n) {
// 1. Let c be the code unit in S at index i.
var c = s.charCodeAt(i)
// 2. Depending on the value of c:
// c < 0xD800 or c > 0xDFFF
if (c < 0xD800 || c > 0xDFFF) {
// Append to U the Unicode character with code point c.
u.push(c)
}
// 0xDC00 ≤ c ≤ 0xDFFF
else if (0xDC00 <= c && c <= 0xDFFF) {
// Append to U a U+FFFD REPLACEMENT CHARACTER.
u.push(0xFFFD)
}
// 0xD800 ≤ c ≤ 0xDBFF
else if (0xD800 <= c && c <= 0xDBFF) {
// 1. If i = n1, then append to U a U+FFFD REPLACEMENT
// CHARACTER.
if (i === n - 1) {
u.push(0xFFFD)
}
// 2. Otherwise, i < n1:
else {
// 1. Let d be the code unit in S at index i+1.
var d = s.charCodeAt(i + 1)
// 2. If 0xDC00 ≤ d ≤ 0xDFFF, then:
if (0xDC00 <= d && d <= 0xDFFF) {
// 1. Let a be c & 0x3FF.
var a = c & 0x3FF
// 2. Let b be d & 0x3FF.
var b = d & 0x3FF
// 3. Append to U the Unicode character with code point
// 2^16+2^10*a+b.
u.push(0x10000 + (a << 10) + b)
// 4. Set i to i+1.
i += 1
}
// 3. Otherwise, d < 0xDC00 or d > 0xDFFF. Append to U a
// U+FFFD REPLACEMENT CHARACTER.
else {
u.push(0xFFFD)
}
}
}
// 3. Set i to i+1.
i += 1
}
// 6. Return U.
return u
}
/**
* @param {!Array.<number>} code_points Array of code points.
* @return {string} string String of UTF-16 code units.
*/
export function codePointsToString(code_points) {
var s = ''
for (var i = 0; i < code_points.length; ++i) {
var cp = code_points[i]
if (cp <= 0xFFFF) {
s += String.fromCharCode(cp)
} else {
cp -= 0x10000
s += String.fromCharCode((cp >> 10) + 0xD800,
(cp & 0x3FF) + 0xDC00)
}
}
return s
}
/**
* @param {boolean} fatal If true, decoding errors raise an exception.
* @param {number=} opt_code_point Override the standard fallback code point.
* @return The code point to insert on a decoding error.
*/
export function decoderError(fatal, opt_code_point) {
if (fatal)
throw TypeError('Decoder error')
return opt_code_point || 0xFFFD
}
/**
* @param {number} code_point The code point that could not be encoded.
* @return {number} Always throws, no value is actually returned.
*/
export function encoderError(code_point) {
throw TypeError('The code point ' + code_point + ' could not be encoded.')
}
/**
* @param {number} code_unit
* @param {boolean} utf16be
*/
export function convertCodeUnitToBytes(code_unit, utf16be) {
// 1. Let byte1 be code unit >> 8.
const byte1 = code_unit >> 8
// 2. Let byte2 be code unit & 0x00FF.
const byte2 = code_unit & 0x00FF
// 3. Then return the bytes in order:
// utf-16be flag is set: byte1, then byte2.
if (utf16be)
return [byte1, byte2]
// utf-16be flag is unset: byte2, then byte1.
return [byte2, byte1]
}
//
// 4. Terminology
//
/**
* An ASCII byte is a byte in the range 0x00 to 0x7F, inclusive.
* @param {number} a The number to test.
* @return {boolean} True if a is in the range 0x00 to 0x7F, inclusive.
*/
export function isASCIIByte(a) {
return 0x00 <= a && a <= 0x7F
}
/**
* An ASCII code point is a code point in the range U+0000 to
* U+007F, inclusive.
*/
export const isASCIICodePoint = isASCIIByte
/**
* End-of-stream is a special token that signifies no more tokens are in the stream.
*/
export const end_of_stream = -1
export const finished = -1
+32
View File
@@ -0,0 +1,32 @@
/* typal types/index.xml externs */
/** @const */
var _textDecoding = {}
/**
* Options for the program.
* @typedef {{ shouldRun: (boolean|undefined), text: string }}
*/
_textDecoding.Config
/** @interface */
function Decoder() {}
Decoder.prototype = {
/**
* @param {Stream} stream The stream of bytes being decoded.
* @param {number} bite The next byte read from the stream.
* @return {?(number|!Array.<number>)} The next code point(s)
* decoded, or null if not enough data exists in the input
* stream to decode a complete code point, or |finished|.
*/
handler: function(stream, bite) {},
}
/** @interface */
function Encoder() {}
Encoder.prototype = {
/**
* @param {Stream} stream The stream of code points being encoded.
* @param {number} code_point Next code point read from the stream.
* @return {(number|!Array.<number>)} Byte(s) to emit, or |finished|.
*/
handler: function(stream, code_point) {},
}