/usr/bin/clickhouse-obfuscator

Simple tool for table data obfuscation.

It reads input table and produces output table, that retain some properties of input, but contains different data.
It allows to publish almost real production data for usage in benchmarks.

It is designed to retain the following properties of data:
- cardinalities of values (number of distinct values) for every column and for every tuple of columns;
- conditional cardinalities: number of distinct values of one column under condition on value of another column;
- probability distributions of absolute value of integers; sign of signed integers; exponent and sign for floats;
- probability distributions of length of strings;
- probability of zero values of numbers; empty strings and arrays, NULLs;
- data compression ratio when compressed with LZ77 and entropy family of codecs;
- continuity (magnitude of difference) of time values across table; continuity of floating point values.
- date component of DateTime values;
- UTF-8 validity of string values;
- string values continue to look somewhat natural.

Most of the properties above are viable for performance testing:
- reading data, filtering, aggregation and sorting will work at almost the same speed
    as on original data due to saved cardinalities, magnitudes, compression ratios, etc.

It works in deterministic fashion: you define a seed value and transform is totally determined by input data and by seed.
Some transforms are one to one and could be reversed, so you need to have large enough seed and keep it in secret.

It use some cryptographic primitives to transform data, but from the cryptographic point of view,
    it doesn't do anything properly and you should never consider the result as secure, unless you have other reasons for it.

It may retain some data you don't want to publish.

It always leave numbers 0, 1, -1 as is. Also it leaves dates, lengths of arrays and null flags exactly as in source data.
For example, you have a column IsMobile in your table with values 0 and 1. In transformed data, it will have the same value.
So, the user will be able to count exact ratio of mobile traffic.

Another example, suppose you have some private data in your table, like user email and you don't want to publish any single email address.
If your table is large enough and contain multiple different emails and there is no email that have very high frequency than all others,
    it will perfectly anonymize all data. But if you have small amount of different values in a column, it can possibly reproduce some of them.
And you should take care and look at exact algorithm, how this tool works, and probably fine tune some of it command line parameters.

This tool works fine only with reasonable amount of data (at least 1000s of rows).


Usage: /usr/bin/clickhouse-obfuscator [options] < in > out

Input must be seekable file (it will be read twice).

Options:
  --help            produce help message
  -S [ --structure ] arg
                    structure of the initial table (list of
                    column and type names)
  --input-format arg
                    input format of the initial table data
  --output-format arg
                    default output format
  --seed arg        seed (arbitrary string), must be random
                    string with at least 10 bytes length;
                    note that a seed for each column is
                    derived from this seed and a column
                    name: you can obfuscate data for
                    different tables and as long as you use
                    identical seed and identical column
                    names, the data for corresponding
                    non-text columns for different tables
                    will be transformed in the same way, so
                    the data for different tables can be
                    JOINed after obfuscation
  --limit arg       if specified - stop after generating
                    that number of rows; the limit can be
                    also greater than the number of source
                    dataset - in this case it will process
                    the dataset in a loop more than one
                    time, using different seeds on every
                    iteration, generating result as large
                    as needed
  --silent arg (=0) don't print information messages to
                    stderr
  --save arg        save the models after training to the
                    specified file. You can use --limit 0
                    to skip the generation step. The file
                    is using binary, platform-dependent,
                    opaque serialization format. The model
                    parameters are saved, while the seed is
                    not.
  --load arg        load the models instead of training
                    from the specified file. The table
                    structure must match the saved file.
                    The seed should be specified
                    separately, while other model
                    parameters are loaded.
  --order arg (=5)  order of markov model to generate
                    strings
  --frequency-cutoff arg (=5)
                    frequency cutoff for markov model:
                    remove all buckets with count less than
                    specified
  --num-buckets-cutoff arg (=0)
                    cutoff for number of different possible
                    continuations for a context: remove all
                    histograms with less than specified
                    number of buckets
  --frequency-add arg (=0)
                    add a constant to every count to lower
                    probability distribution skew
  --frequency-desaturate arg (=0)
                    0..1 - move every frequency towards
                    average to lower probability
                    distribution skew
  --determinator-sliding-window-size arg (=8)
                    size of a sliding window in a source
                    string - its hash is used as a seed for
                    RNG in markov model


Example:
    /usr/bin/clickhouse-obfuscator --seed "$(head -c16 /dev/urandom | base64)" --input-format TSV --output-format TSV --structure 'CounterID UInt32, URLDomain String, URL String, SearchPhrase String, Title String' < stats.tsv