REDROOM

Navigation

Breadcrumbs

/ / opt / alt / python37 / share / doc / alt-python37-charset-normalizer

Create

Create File | Create Folder | Upload Files | Run Command

Edit File Size: 10.08 KB

//opt/alt/python37/share/doc/alt-python37-charset-normalizer/README.md

Text Base64

<h1 align="center">Charset Detection, for Everyone 👋 <a href="https://twitter.com/intent/tweet?text=The%20Real%20First%20Universal%20Charset%20%26%20Language%20Detector&url=https://www.github.com/Ousret/charset_normalizer&hashtags=python,encoding,chardet,developers"><img src="https://img.shields.io/twitter/url/http/shields.io.svg?style=social"/></a></h1>

The Real First Universal Charset Detector 
 <a href="https://pypi.org/project/charset-normalizer">
 <img src="https://img.shields.io/pypi/pyversions/charset_normalizer.svg?orange=blue" />
 </a>
 <a href="https://codecov.io/gh/Ousret/charset_normalizer">
 <img src="https://codecov.io/gh/Ousret/charset_normalizer/branch/master/graph/badge.svg" />
 </a>
 <a href="https://pepy.tech/project/charset-normalizer/">
 <img alt="Download Count Total" src="https://pepy.tech/badge/charset-normalizer/month" />
 </a>

> A library that helps you read text from an unknown charset encoding. Motivated by `chardet`,
> I'm trying to resolve the issue by taking a new approach.
> All IANA character set names for which the Python core library provides codecs are supported.

>>>>> <a href="https://charsetnormalizerweb.ousret.now.sh" target="_blank">👉 Try Me Online Now, Then Adopt Me 👈 </a> <<<<<

This project offers you an alternative to **Universal Charset Encoding Detector**, also known as **Chardet**.

| Feature | [Chardet](https://github.com/chardet/chardet) | Charset Normalizer | [cChardet](https://github.com/PyYoshi/cChardet) |
| ------------- | :-------------: | :------------------: | :------------------: |
| `Fast` | ❌ | :heavy_check_mark: | :heavy_check_mark: |
| `Universal**` | ❌ | :heavy_check_mark: | ❌ |
| `Reliable` **without** distinguishable standards | ❌ | :heavy_check_mark: | :heavy_check_mark: |
| `Reliable` **with** distinguishable standards | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| `Free & Open` | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| `License` | LGPL-2.1 | MIT | MPL-1.1
| `Native Python` | :heavy_check_mark: | :heavy_check_mark: | ❌ |
| `Detect spoken language` | ❌ | :heavy_check_mark: | N/A |
| `Supported Encoding` | 30 | :tada: [93](https://charset-normalizer.readthedocs.io/en/latest/user/support.html#supported-encodings) | 40

*\*\* : They are clearly using specific code for a specific encoding even if covering most of used one* 
Did you got there because of the logs? See [https://charset-normalizer.readthedocs.io/en/latest/user/miscellaneous.html](https://charset-normalizer.readthedocs.io/en/latest/user/miscellaneous.html)

## ⭐ Your support

*Fork, test-it, star-it, submit your ideas! We do listen.*
  
## ⚡ Performance

This package offer better performance than its counterpart Chardet. Here are some numbers.

| Package       | Accuracy       | Mean per file (ms) | File per sec (est) |
| ------------- | :-------------: | :------------------: | :------------------: |
|      [chardet](https://github.com/chardet/chardet)        |     92 %     |     220 ms      |       5 file/sec        |
| charset-normalizer |    **98 %**     |     **40 ms**      |       25 file/sec    |

| Package       | 99th percentile       | 95th percentile | 50th percentile |
| ------------- | :-------------: | :------------------: | :------------------: |
|      [chardet](https://github.com/chardet/chardet)        |     1115 ms     |     300 ms      |       27 ms        |
| charset-normalizer |    460 ms     |     240 ms      |       18 ms    |

Chardet's performance on larger file (1MB+) are very poor. Expect huge difference on large payload.

> Stats are generated using 400+ files using default parameters. More details on used files, see GHA workflows.
> And yes, these results might change at any time. The dataset can be updated to include more files.
> The actual delays heavily depends on your CPU capabilities. The factors should remain the same.

[cchardet](https://github.com/PyYoshi/cChardet) is a non-native (cpp binding) and unmaintained faster alternative with 
a better accuracy than chardet but lower than this package. If speed is the most important factor, you should try it.

## ✨ Installation

Using PyPi for latest stable
```sh
pip install charset-normalizer -U
```

If you want a more up-to-date `unicodedata` than the one available in your Python setup.
```sh
pip install charset-normalizer[unicode_backport] -U
```

## 🚀 Basic Usage

### CLI
This package comes with a CLI.

```
usage: normalizer [-h] [-v] [-a] [-n] [-m] [-r] [-f] [-t THRESHOLD]
                  file [file ...]

The Real First Universal Charset Detector. Discover originating encoding used
on text file. Normalize text to unicode.

positional arguments:
  files                 File(s) to be analysed

optional arguments:
 -h, --help show this help message and exit
 -v, --verbose Display complementary information about file if any.
 Stdout will contain logs about the detection process.
 -a, --with-alternative
 Output complementary possibilities if any. Top-level
 JSON WILL be a list.
 -n, --normalize Permit to normalize input file. If not set, program
 does not write anything.
 -m, --minimal Only output the charset detected to STDOUT. Disabling
 JSON output.
 -r, --replace Replace file when trying to normalize it instead of
 creating a new one.
 -f, --force Replace file without asking if you are sure, use this
 flag with caution.
 -t THRESHOLD, --threshold THRESHOLD
 Define a custom maximum amount of chaos allowed in
 decoded content. 0. <= chaos <= 1.
 --version Show version information and exit.
```

```bash
normalizer ./data/sample.1.fr.srt
```

:tada: Since version 1.4.0 the CLI produce easily usable stdout result in JSON format.

```json
{
    "path": "/home/default/projects/charset_normalizer/data/sample.1.fr.srt",
    "encoding": "cp1252",
    "encoding_aliases": [
        "1252",
        "windows_1252"
    ],
    "alternative_encodings": [
        "cp1254",
        "cp1256",
        "cp1258",
        "iso8859_14",
        "iso8859_15",
        "iso8859_16",
        "iso8859_3",
        "iso8859_9",
        "latin_1",
        "mbcs"
    ],
    "language": "French",
    "alphabets": [
        "Basic Latin",
        "Latin-1 Supplement"
    ],
    "has_sig_or_bom": false,
    "chaos": 0.149,
    "coherence": 97.152,
    "unicode_path": null,
    "is_preferred": true
}
```

### Python
*Just print out normalized text*
```python
from charset_normalizer import from_path

results = from_path('./my_subtitle.srt')

print(str(results.best()))
```

*Normalize any text file*
```python
from charset_normalizer import normalize
try:
    normalize('./my_subtitle.srt') # should write to disk my_subtitle-***.srt
except IOError as e:
    print('Sadly, we are unable to perform charset normalization.', str(e))
```

*Upgrade your code without effort*
```python
from charset_normalizer import detect
```

The above code will behave the same as **chardet**. We ensure that we offer the best (reasonable) BC result possible.

See the docs for advanced usage : [readthedocs.io](https://charset-normalizer.readthedocs.io/en/latest/)

## 😇 Why

When I started using Chardet, I noticed that it was not suited to my expectations, and I wanted to propose a
reliable alternative using a completely different method. Also! I never back down on a good challenge!

I **don't care** about the **originating charset** encoding, because **two different tables** can
produce **two identical rendered string.**
What I want is to get readable text, the best I can.

In a way, **I'm brute forcing text decoding.** How cool is that ? 😎

Don't confuse package **ftfy** with charset-normalizer or chardet. ftfy goal is to repair unicode string whereas charset-normalizer to convert raw file in unknown encoding to unicode.

## 🍰 How

- Discard all charset encoding table that could not fit the binary content.
  - Measure chaos, or the mess once opened (by chunks) with a corresponding charset encoding.
  - Extract matches with the lowest mess detected.
  - Additionally, we measure coherence / probe for a language.

**Wait a minute**, what is chaos/mess and coherence according to **YOU ?**

*Chaos :* I opened hundred of text files, **written by humans**, with the wrong encoding table. **I observed**, then
**I established** some ground rules about **what is obvious** when **it seems like** a mess.
 I know that my interpretation of what is chaotic is very subjective, feel free to contribute in order to
 improve or rewrite it.

*Coherence :* For each language there is on earth, we have computed ranked letter appearance occurrences (the best we can). So I thought
that intel is worth something here. So I use those records against decoded text to check if I can detect intelligent design.

## ⚡ Known limitations

- Language detection is unreliable when text contains two or more languages sharing identical letters. (eg. HTML (english tags) + Turkish content (Sharing Latin characters))
  - Every charset detector heavily depends on sufficient content. In common cases, do not bother run detection on very tiny content.

## 👤 Contributing

Contributions, issues and feature requests are very much welcome. 
Feel free to check [issues page](https://github.com/ousret/charset_normalizer/issues) if you want to contribute.

## 📝 License

Exit & Reset Text mode: syntax highlighting auto-detects file type.

Directory Contents

Dirs: 0 × Files: 3

Sort:

Enable drag-to-move

Name	Size	Perms	Modified	Actions
CHANGELOG.md	9.99 KB	lrw-r--r--	2022-02-12 14:32:58	Edit Download recursive recursive
LICENSE	1.04 KB	lrw-r--r--	2022-02-12 14:32:58	Edit Download recursive recursive
README.md	10.08 KB	lrw-r--r--	2022-02-12 14:32:58	Edit Download recursive recursive