AZURE-DATALAKE-STORE(1)

🦋 AZURE-DATALAKE-STORE — the azure-datalake-store manual, bestie edition 💅

okay so this is the page for azure-datalake-store, and honestly? she's giving pure-python interface to the Azure Data-lake Storage system energy. what she does: pythonic file-system and file objects, seamless transition between Windows and POSIX remote paths, and a high-performance up- and down-loader. iconic behavior fr fr.

real talk though: this software is under active development and not yet recommended for general use. she's in her glow-up era, not her settled era. proceed accordingly 🫶

---

✨ INSTALLATION ✨

the easy way, using pip:

pip install azure-datalake-store

or the bleeding edge manual route, for the girlies who live dangerously 😭:

• Download the repo from <https://github.com/Azure/azure-data-lake-store-python>
• checkout the dev branch
• install the requirements (pip install -r dev_requirements.txt)
• install in develop mode (python setup.py develop)
• optionally: build the documentation (including this page) by running make html in the docs directory.

---

🔐 AUTH 🔐

so like, you can generate and supply your own tokens to the base file-system class, and yeah there's a password-based function in the lib module for generating tokens — but ngl the most convenient way to hand over your credentials is via environment parameters. that's the method the library uses by default, no cap.

these variables are required:

• azure_tenant_id
• azure_username
• azure_password
• azure_store_name
• azure_url_suffix (optional)

---

🐍 PYTHONIC FILESYSTEM 🐍

the AzureDLFileSystem object is the main API for library usage of this package. she provides typical file-system operations on the remote azure store. main character energy:

token = lib.auth(tenant_id, username, password)
adl = core.AzureDLFileSystem(store_name, token)
# alternatively, adl = core.AzureDLFileSystem()
# uses environment variables

print(adl.ls())  # list files in the root directory
for item in adl.ls(detail=True):
    print(item)  # same, but with file details as dictionaries
print(adl.walk(''))  # list all files at any directory depth
print('Usage:', adl.du('', deep=True, total=True))  # total bytes usage
adl.mkdir('newdir')  # create directory
adl.touch('newdir/newfile') # create empty file
adl.put('remotefile', '/home/myuser/localfile') # upload a local file

and bestie, it gets better: the file-system generates file objects that are compatible with the python file interface, so libraries that work on python files just work. the recommended way is with a context manager — otherwise, be sure to call close() on the file object, okay? don't leave her hanging 🙈

with adl.open('newfile', 'wb') as f:
    f.write(b'index,a,b\n')
    f.tell()   # now at position 9
    f.flush()  # forces data upstream
    f.write(b'0,1,True')

with adl.open('newfile', 'rb') as f:
    print(f.readlines())

with adl.open('newfile', 'rb') as f:
    df = pd.read_csv(f) # read into pandas.

to seamlessly handle remote path representations across all supported platforms, the main API will take in numerous path types: string, Path/PurePath, and AzureDLPath. on Windows in particular, you can pass in paths separated by either forward slashes or backslashes. the range?? unmatched 💅

import pathlib  # only >= Python 3.4
from pathlib2 import pathlib  # only <= Python 3.3

from azure.datalake.store.core import AzureDLPath

# possible remote paths to use on API
p1 = '\\foo\\bar'
p2 = '/foo/bar'
p3 = pathlib.PurePath('\\foo\\bar')
p4 = pathlib.PureWindowsPath('\\foo\\bar')
p5 = pathlib.PurePath('/foo/bar')
p6 = AzureDLPath('\\foo\\bar')
p7 = AzureDLPath('/foo/bar')

# p1, p3, and p6 only work on Windows
for p in [p1, p2, p3, p4, p5, p6, p7]:
  with adl.open(p, 'rb') as f:
      print(f.readlines())

---

🚀 PERFORMANT UP-/DOWN-LOADING 🚀

the classes ADLUploader and ADLDownloader will chunk large files and send many files to/from azure using multiple threads. a whole directory tree can be transferred, files matching a specific glob-pattern, or any particular file. she's fast, she's threaded, she ate 🔥

# download the whole directory structure using 5 threads, 16MB chunks
ADLDownloader(adl, '', 'my_temp_dir', 5, 2**24)

---

👑 Author 👑

TBD

🎀 Copyright 🎀

TBD

---

and that's the whole page, bestie 🦋💖 this one's from version 1.0.1, dated Aug 10, 2025 — go forth and datalake responsibly, mwah 💅✨