Changelogs

1.7 - Jan 25th, 2025

Cli:

sql modifications:

  • Added lots of caching mechanisms: s[“db”], db[“table”] and table[“id”]

  • Allowed setting json columns using dict

  • Added option to automatically html escape for text fields, to avoid cross site scripting

Generic:

1.6 - Mar 8th, 2024

Cli:

Generic:

  • Added monkey patched methods os.netstat() and os.killPort()

  • Added dollar()

  • Added several language formatters like js, py, html, sql

  • Added kcom for external hardware communications, with Gsm and Host

  • Added monkey-patched method torch.transpose_axes() to match numpy’s signature

  • Added monkey-patched method plt.worldmap() to quickly plot out a world map

  • Added monkey-patched method html.b64escape() to provide a more robust html escaping where JS is available

  • Added monkey-patched method pandas.DataFrame.newColName()

  • Added module kop for optics simulation

  • Added module zircon for browser automation

1.5 - Jan 6th, 2024

Cli:

Generic:

  • Added websocket client and server support at kws

  • Added logging integration with https://logs.mlexps.com using :meth`~k1lib.log`

  • Added module kast for advanced abstract syntax tree manipulation in support for the JS transpiler

  • Added resolve() to resolve coroutines everywhere

  • Added serve.json(), serve.date(), serve.serialized(), serve.apiKey() and souped up serve a little

  • Added serpent, a lua serialization module

  • Added formatting function rmAnsi()

1.4 - June 30th, 2023

Cli:

  • Added ability for an existing cli to join with a function that’s not a cli transparently. Meaning this is now possible: 3 | (iden() | (lambda x: x**4))

  • Added toHash, toStd

  • Accelerated balanceFolder()

  • Accelerated repeat, repeatFrom, joinStreams and unique for array types

  • Added library-wide optimization that allows np.random.randn(3, 4, 5, 6) | item().all() | shape() == (3, 5, 6) to operate on the array directly and not be broken up by .all()

  • Removed structural.accumulate and structural.indexTable, as it can be done using groupBy and it’s not used enough

  • Added sketch, syncStepper

  • Added alias T

  • Added Cython support and cpu limit settings option for applyCl

  • Added full distribution mode to intersection

  • Added builtin ML models support: embed, complete, kmeans, tsne and bloom Model operations are also optimized to batch things together automatically

  • Added new way to deal with infinite iterators generated by repeat and friends

  • Added multiple table pretty-format mode to pretty

  • Removed kcsv, because it’s ridiculous to have a single short method in a big module and it’s not intuitive. Replaced that with toCsv

  • Added ktree for dealing with tree structures

  • Added support for .zip files for kunzip, ls() and cat()

  • Added support for custom datatypes for ls() and cat()

  • Added k1lib.cli.lsext, for ls’s custom datatypes (“ls extensions”), particularly MySQL, PostgreSQL and SQLite

  • Added robust Actor model for applyCl.

  • Accelerated rows and cut for array types

  • Added unsort mode to sort

  • Added roll, batchify()

  • Added k1lib.cli.kapi to use ML apis published on mlexps.com

  • Added k1lib.selen to extract main content of a webpage using Selenium

Generic:

1.3 - May 26th, 2023

Cli:

  • Revamped docs a lot

  • Added applyCl to execute functions across an entire cluster

  • Auto get representation if not string in grep.

  • Added aliases map_, filter_ and flatten

  • Added string execution 3 | aS("x+2") capability to all clis

  • Added toHtml to quickly convert PIL image to html tag

  • Accelerated filt for array types even more

  • Added ungroup, and added more modes to groupBy

  • Fixed cut()[2:-2] edge case

  • Added rItem and urlPath for quality of life

  • Added demiliter-splitting mode to toDict

  • Added nb.execute.rightAway(), toAscii(), backup, kzip and kunzip, resume()

  • Added modifier.applyMp.shared() to easily share memory between processes

  • Accelerated cat() using k1a if sB and eB are specified

  • Added multi-column mode to apply

  • Removed inp.curl() because you can do cat("http://example.com") directly

  • Added array mode and retries parameter to tryout

  • Allowed building pipelines with normal classes that only define __ror__ but not subclass BaseCli

Generic:

  • Revamped serve a little, adding a more robust type system

  • Added more default imports like ray and pyarrow

  • Added Scroll and Toggle

  • Added row(), col(), pre() and h() to quickly build html strings

  • Added capturePlt() to capture matplotlib plots

  • Added formatting functions fromSize() and colors()

  • Added pipe mode to Carousel and Wrapper

1.2 - Feb 7th, 2023

Cli:

Generic:

1.1 - Oct 3rd, 2022

Cli:

  • Added dictionary mode, with table name to tables()

  • Added records(), toGray, plotImgs, toBytes, indexTable, splitC

  • k1a speedups: cat

  • Fixed strange bug in applyMp that happens between PyTorch and multiprocessing

  • Added tensor/bytes/figure to image feature to toPIL()

  • Added image saving mode to file()

  • Added separator mode to oneHot

  • Removed conv.toStr, conv.toSet, conv.toIter, conv.toBin, conv.toIdx, conv.toDictF, modifier.applySerial because they’re kinda meaningless and redundant.

  • Added bytes mode chunking to cat

  • Removed module k1lib.cli.others as it’s pretty useless

Generic:

  • Added pushNotification(), fmt.throughput()

  • Removed clearLine(), close(), textToHtml() as they are quite niche

  • Moved k1lib.sameStorage() to torch.sameStorage()

  • Monkey patched pandas.core.frame.DataFrame.table(), builtins.str.splitCamel

  • Added intersection operation to Domain

  • Removed library’s PyTorch dependency, so that it can be lightweight enough to be used inside containers

Deep learning:

1.0 - Aug 9th, 2022

Cli:

  • Removed conv.toNumpy, as toList() | aS(np.array) does the same thing

  • Added fill mode to lookup

  • Changed tee style to be more natural

  • Removed utils.headerIdx, as insertIdColumn() roughly does the same thing

  • Removed init.manyToMany, replaced with apply to simplify things

  • Removed unstable op in feature and added combined compare ops to it

  • Added bounds comparison to op, thus removed filt.inRange

  • Added every mode to tee.

  • Simplified insertColumn

  • Added oneHot.

  • Restructured expand argument feature in applySerial

  • Added cif module

  • Removed entrez module, as it doesn’t add much value

  • Simplified file by removing explicit text/byte mode param

LLVM:

Generic:

  • Added library k1a to speed up certain parts of the main library by compiling things down to C.

  • Added k1ui support within k1lib. It’s a Java program that can communicate over the network and provides apis to manipulate mouse/keyboard and screens.

DL:

0.17 - Jul 15th, 2022

Cli:

  • Accelerated head for all sliceable types

  • Add fractional mode to head

  • Added expanded arguments mode to apply

  • Added column mode to sortF

  • Added gb.feats.root(), gb.feats.tags()

  • Removed gb.feats.tag(), use gb.feats.tags() instead

  • Added predicate mode to grep

  • Added dictFields

  • Accelerated groupBy significantly

  • Removed collate(), as there’s no use for it

  • Accelerated shape, thus removing conv.lengths and conv.toLens as shape(0).all() is just as good

0.16 - Jun 28th, 2022

Cli:

  • Accelerated head if input’s a torch.Tensor or np.ndarray.

  • Added tree(), timeLimit, split, lookup.

  • Renamed joinList to insert, cause that’s more intuitive

  • Removed structural.insertRow, as insertRow(1, 2) can be achieved using ~insert(1, 2)

  • Souped up insertColumn

  • Added column mode in grep.

  • Added JIT compilation for op

  • Added alias parallel and toLens

  • Accelerated batched if input’s a range

  • Added accelerated mode to cat()

  • Grouped to__ clis into the module conv.

  • Removed redundant toType and identity

0.15 - Mar 22nd, 2022

Generic:

  • Added UValue, now()

  • Patched numpy.ndarray’s __or__ method to work better with cli tools

  • Added a few mathematical and physics constants in k1lib.imports

  • Added selector.ModuleSelector.cutOff()

  • Renamed k1lib.integrate() to k1lib.integral(), to avoid conflict with existing cli

  • Added sameStorage()

Cli:

0.14 - May 1st, 2022

Generic:

Cli:

  • Changed implementation of rows. Should be much more stable and perdictable than the last version

  • Fixed transpose so that it works efficiently with numpy arrays

  • Fixed splitW so that it includes dangling elements at the end

  • Revamped k1lib.cli.kxml heavily

  • Added structural.count.join(), to join multiple count results together

  • Added aliases toImg, aS

  • Added toRgb and toRgba.

  • Added clipboard and its dependency requirement.

  • Fixed bug in accumulate.

  • Removed modifier.replace() and modifier.remove(), as those can be done by op just fine.

  • Added seed option in randomize

  • Added window.

0.13 - Mar 23rd, 2022

Generic:

Cli:

0.12 - Dec 1st, 2021

Generic:

  • Disallow adding new settings accidentally when using context().

Cli:

  • Fixed cmd piping mechanism into each other.

  • Added quiet setting, used by cmd.

  • Made columns more robust so that it can deal with missing elements and longer rows below.

  • Added operator - to filt for extra intuitiveness.6

  • Added bindec, toBin, toIdx, sam.flag, longFa() and idx.

  • Moved sam.quality to bio.quality(), and added cli.bio.phred, cli.sam.header settings.

  • Moved setting cli.oboFile to cli.bio.go, and cli.lookupImgs to cli.bio.lookupImgs.

  • Added fill param to item.

  • Optimizes higher order operations quite a bit, and added setting cli.atomic.

  • Renamed init.manyToManySpecific to init.mtmS, and added method f().

  • Added fill() and wrap() to transpose().

  • Revamped how cmd works, allowed real time output yielding, added bytes mode and option on how to return stdout and stderr.

0.11 - Nov 24th, 2021

DL:

Cli:

  • Added multidimensional mode to transpose.

  • Changed param position of deref so that it’s more convenient.

  • Moved inp.toPIL to utils.toPIL, to be more consistent.

  • Added nb. As a consequence, removed nbCells() and executeNb().

  • Added temporary file and append features to file().

  • Added immediate resolve option to cmd.

  • Added N option (max sections) and changed till() mechanics in grep. Also added sep option and as a consequence, removed grepToTable.

  • Added home directory support for file, cat() and ls().

0.10 - Nov 17th, 2021

DL:

  • Added nbCells(), timer()

  • Changed Absorber methods so that it won’t use special words like int, float, len, str and added ab_contains() and ab_fastF().

  • Revamped lots of Callbacks docs

  • Renamed CaptureStdout to captureStdout, to match context manager’s styles.

Cli:

  • Added utilization parameter to applyMp.

  • Added fastF()

  • Added tensor mode to mask.

  • Added filt and apply modes with columns to trace.

  • Made deref so that it ignores dictionaries.

  • Added some compatibility with numpy arrays across clis.

  • Added cacheLimit parameter to apply and as a consequence, removed applyCached.

  • Added toType and applyTh.

  • Added bs parameter to applyMp and as a consequence, removed applyMpBatched().

0.9 - Mov 14th, 2021

DL:

Cli:

  • Added kwargs argument to modifier.applyMpBatched().

  • Added RunOnce, MaxDepth.

  • Added alias iden.

  • Added trace() mechanism, with accompanying tutorial.

  • Remove shape’s tensor special case

  • Added applySerial.

  • Added inf and context to cliSettings.

  • Fixed bug where "abc" | deref() will actually split the string into characters.

  • Terminates pool in applyMp if is keyboard interrupted, and in general made it much more robust.

  • Stops deref early if encountered yieldSentinel

  • Optimizes op even further, thus removing isValue(), as filt(op() == value) does the same thing.

0.8 - Nov 7th, 2021

DL:

Cli:

  • Modified shape so that it can detect multiple dimensions now, not just 2.

  • Added extend(), fullness().

  • Added applyCached.

  • Brought back table().

  • Removed startswith() and endswith(), as filt(op().startswith()) does the same thing.

0.7 - Nov 3rd, 2021

DL:

  • Added metab as extra metadata from dataloaders to Learner in the core training loop and modified Recorder to return it if available.

  • Added n option to AutoIncrement.

  • Added mask().

  • Removed k1lib.stats(), replaced it with torch.Tensor.stats() directly

  • Added op option to Fn.

  • Added torch.Tensor.hasInfs()

  • Made LossNLLCross integrates with AccF instead.

  • Made AccF integrates with ConfusionMatrix.

  • Made ConfusionMatrix resilient against nans and infs, and fixed norm strategy (tensor rows divide by max rows, instead of tensor columns divide by max rows, like before).

Cli:

  • Added mask, unsqueeze().

  • Made op compatible with lots of other cli tools, so you can do stuff like filt(op() > 3, 1).

  • Removed split, as op().split().all() | joinStreams() does the same thing. This is longer, so I kinda want to put it back in, but the name is incredibly oxymoronic.

  • Removed listToTable() and its alias tableFromList(), as they are too simple, and the new unsqueeze(1) does the same thing.

  • Removed args option in applyMp, as it feels clunky.

  • Removed cats as cat().all() does the same job.

  • Sped up op to have roughly same performance as apply. As a consequence, removed [lstrip, rstrip, strip, upper, lower] clis as stuff like op().upper().all() does the same and I want to simplify the collection of cli tools.

  • Completely removed k1lib.cli.ctx module, as the performance sacrifices are too much, and it doesn’t add a lot of useful features.

  • Renamed utils.to1Str to join as that is more intuitive.

  • Removed structural.table, as op().split().all() does the same thing

0.6 - Oct 30th, 2021

DL:

  • Added LinBlock, Absorber.

  • Added __module__ impersonation for wrapMod.

  • Renamed LossLambda to LossF

  • Renamed torch.nn.Module.preserveDevice() to torch.nn.Module.deviceContext() and add preserve buffers capability.

  • Added torch.nn.Module.gradContext().

k1lib.selector.ModuleSelector:

Cli:

  • Added raiseError option in intercept.

  • Added multi argument capabilities to __call__().

  • Added op.

  • Changed default param ignoreTensors of deref to True.

  • Ignores Module by default in deref.

0.5 - Oct 27th, 2021

DL:

Cli:

  • Added chained mode to item

  • Fixed ignoreTensor mechanic in deref

  • Added prefetch mode in applyMp

  • Added applyMpBatched that executes lots of transformations in a single job.

  • Added clearPools() and pools().

  • Added bytes reading and writing to cat file.

  • Added activeSamples.

  • Removed aliases splitColumns and joinColumns.

  • Removed stitch, as to1Str().all() does the same thing.

0.4 - Oct 23rd, 2021

This update moves things around a lot. The actual coding experience stays the same, but a lot of links < 0.4 will be broken in 0.4 docs. Go to 0.3 docs if you wish to follow links.

Molecule (k1lib.mo):

  • Added close coulomb force calculations to simulator to make it more realistic (cyclohexane chair config possible now!)

  • Added optional Hydrogen viewing to various functions in mo

  • Fixed parsing bug where it doesn’t recognize “3-methyl” group in “3-methylpentane”.

DL:

  • Don’t let AutoIncrement’s value property to auto increment internal value.

  • Removed CyclicRandomSampler, as range(n) | repeatFrom() | randomize(n)` does the same thing.

  • Removed FunctionDataset, added simpler FunctionData as replacement.

  • Removed DataLoader, Data and DatasetWithSampler, as they’re complex and obsolete.

  • Added tfImg(), tfFloat(), analyzeFloat()

  • Monkey-patched torch.nn.Module to have piping capabilities.

  • Renamed k1lib.data to k1lib.kdata, and exposed it automatically in imports.

  • Renamed k1lib.nn to k1lib.knn, and exposed it automatically.

Bioinfo cli:

  • Renamed k1lib.bioinfo.cli to k1lib.cli

  • Added stagger() as the new interface for dataloaders.

  • Added intercept, toPIL.

  • Added PIL conversion mode to utils.toTensor

  • Added Tensor shape mode in utils.shape.

  • Added others module, with crissCross()

  • Added Tensor mode to toSum, toAvg, toMin, toMax and oneToMany.

  • Added alias toMean.

  • Added dtype choice to toTensor.

0.3 - Oct 18th, 2021

Background stuff:

  • Added mo tutorial

DL:

Bioinfo cli:

  • Removed notIn, as inSet does the job just as well

  • Included cli import command in imports by default

  • Modified toFloat() and modifier.toInt() so they can force weird values to 0.

  • Moved existing toDict to toDictF, and created a new toDict.

0.2 - Sep 24th, 2021

Pretty minor update. I just want to push this out as soon as possible

Background stuff:

  • Changed versioning system so that it makes more sense

  • Added osic and mp tutorials

  • Added more speed analysis in the cli tutorial

Bioinfo cli:

  • Removed dirs and files parameters in ls, as isFile is good enough

  • Disable module ctx by default, to improve performance

  • Replaced nonEmptyStream to empty

0.1.13 - Sep 20th, 2021

Background stuff:

  • Removed walrus operator library-wide, so that it can be used with Python < 3.8

Bioinfo cli:

  • Added breakIf, isFile

  • Replaced infiniteF with repeatF

  • Replaced infiniteFrom with repeatFrom

  • Replaced dereference with deref

  • Replaced ctx.dereference with ctx.deref

  • Modified applyMp to use dill to deserialize everything

  • Modified to1Str so that it returns a string, instead of yielding

  • Modified toStr so that it can accept an optional column

  • Modified splitList so that it accepts *args

  • Removed applySingle, but retain applyS

DL:

0.1.12 - Sep 3rd, 2021

Background stuff:

  • Removed docs’s generated fonts folder, as it takes 8MB and isn’t even used. Should now be sustainable for 100 versions on github pages.

Bioinfo cli:

  • Removed wrapping __ror__ operator if __init__ is not defined, to make things performant.

  • Added speed analysis in the cli tutorial.

  • Added init.BaseCli.__call__(), consume, ctx.ctx(), ctx.f, peekF, expandE, utils.dereference.__invert__(), unique, notIn, sortF, toMax, toMin, union, ls(), instanceOf(), randomize

  • Removed ctx.identity

  • Added shortcuts to ctx.setC() and ctx.getC()

  • Made headerIdx() context-aware

  • Put ctx.Promise dereferencing at init.BaseCli.__ror__(), instead of dynamic patching at k1lib.bioinfo.cli construction time

  • Get rid of stdout style inconsistencies. Previously, it looks like [1, 2] | stdout, now it looks like [1, 2] | stdout().

  • Added multiprocessing capabilities with applyMp

  • Added maxDepth option for dereference

  • Added includeLast option to batched

  • Added float("inf") option to bs parameter of batched

  • Fixed permute to be able to take in generator for rows.

  • Fixed stdout to be able to display non-iterable inputs

  • Fixed dereference so that it will handle poking errors

  • Fixed count so that it can deal with list. Used to throw unhashable type: ‘list’

  • Added more flexibility with toFloat(), toInt() and sort

DL:

  • Added classes AutoIncrement, Wrapper

  • Added functions positionalEncode(), debounce()

  • Fixed executeNb’s _globals doc-backref problem. Also clears plot automatically after executing a cell now.

0.1.11 - Aug 25th, 2021

Background stuff:

  • Added testing section for unit tests (simple assert statements inside the notebooks)

Bioinfo cli:

  • Added numpy/torch checks to dereference

  • Renamed sample to peek

  • Added infiniteFrom, joinStreamsRandom, batched, collate, splitList

  • Added infiniteF

  • Added toTensor

  • Replaced infinite in favor of repeat

  • Updated all cli tools to use super().__init__()

  • Added ctx module, with Promise, enum, identity classes, and getC(), setC() methods

DL:

  • Added k1lib.data.CyclicRandomSampler and k1lib.data.DatasetWithSampler

  • Added highlight feature in callbacks.Callbacks.checkpointGraph()

  • Added k1lib.viz.Carousel.pop(), k1lib.viz.Carousel.__getitem__()

  • Added k1lib.Learner.sample()

  • Disallow k1lib.executeNb() to execute lines that starts with “!”, also removed parameter catchErrors, as it’s quite useless

  • Added k1lib.Object()

0.1.10 - Aug 22nd, 2021

Background stuff:

  • Added notebook regression tests

  • Added tutorials automated building tool

  • Added k1lib.ignoreWarnings

  • Tutorials part moved to official docs (rather than .md files on github)

  • Added covid tutorial

Bioinfo cli:

  • Reintroduced permute, as rows is inadequate

  • Added lengths, applySingle

  • Exposed serial, oneToMany, manyToMany, and manyToManySpecific

DL:

0.1.9 - Aug 19th, 2021

Bioinfo cli:

  • Added k1lib.Domain class

  • Added __invert__ option for rows and columns by incorporating k1lib.Domain

  • Sped up rows significantly

  • Removed cli tool every as rows is good enough

  • Added __invert__ option for head. As a result, removed nhead

  • Added till() mechanism to grep cli tool

  • Added gb cli module to deal with genbank file format

  • Added tableFromList() as alias of listToTable()

  • Renamed avg to toAvg

  • Added toSum

DL:

0.1.8 - Aug 15th, 2021

  • Fixed installation bug where readme.md would be absent, and setup.py won’t work at all.

Bioinfo cli:

  • Huge revamp of how tables are treated. Before, row elements are in a single string, separated by a delimiter. Now each row is a full fledged list

  • Added grepToTable, joinList, contains, table, stitch, listToTable, insertColumn cli tools

  • Removed joinRows but still keep joinStreams as “joinRows” is not really obvious

  • Removed permute as columns is pretty much identical

  • Renamed inside to inSet

  • Cut out nested functionality from grep to grepToTable

0.1.7 - Aug 14th, 2021

Interface with external modules:

Callbacks:

  • Changed Callbacks.suspend() to take in names only

  • Added Callbacks.suspendClasses() to take in classes

  • Changed Callbacks.suspendEvaluation() to Callbacks.suspendEval(), also added functionality for white and black lists

  • Ripped out complex Callback.__getattr__() mechanism (that automatically searches the parent learner for missing attributes) to improve speed

  • Added k1lib.callbacks.core callbacks CoreNormal and CoreRNN

  • Added more plot decorators for MemoryProfiler, including backward-forward separator with their text

  • Changed LossLandscape callback so that it follows the common flow structure. Before, it implements a whole new cbs(...) schema that essentially duplicated the loop inside of k1lib.Learner

Learner:

  • Implemented grace stop handling for Learner. Before, if someone throws CancelRunException while executing a batch, then it will immediately jump to the end of the run. Changed it so that it will be caught in the batch loop, execute checkpoints cancelBatch and endBatch and rethrow it. Same thing with CancelEpochException

  • Moved loss function from being a core functionality inside Learner to an external callback, so that adapting multiple loss functions are possible

  • Added Learner.__call__()

Bioinfo cli:

  • Added isValue, intersection, AA_, infinite, dereference cli tools

  • Modified cat to be able to take in file name by pipe

  • Added new joining operator “+” between init.BaseCli

Style changes:

0.1.6 - Aug 13th, 2021

I don’t really keep track of things 0.1.6 and before, so nothing here really.