Querying HTML, Media, and Storage

Web extraction, media processing, and storage formats all benefit from working with structured representations rather than treating bytes as opaque text.

Parse HTML Before Querying It #

XPath operates on a document tree, not a raw HTML string. Parse web content with an HTML parser, then query the resulting DOM. Browser DOMs, HTML parsers, and XML parsers can repair malformed markup differently, so use the same parser class in tests and production.

CSS selectors are concise when the document author controls stable IDs and classes. XPath can query text and navigate relative to matched nodes, which makes it suitable for document analysis and crawlers.

//article//h2
//a[@rel='next']/@href
//table//tr[td]
//*[contains(concat(' ', normalize-space(@class), ' '), ' product ')]
//h2[normalize-space(.)='Specifications']/following-sibling::ul[1]/li

// searches descendants, @name selects an attribute, predicates filter a node set, and an axis such as following-sibling navigates from a selected node. The class expression matches one token rather than accidentally matching product-card for a query that requires product.

Avoid positional paths such as //div[7] unless the structure is contractual. Anchor on a semantic element, an accessible name, a heading, a link relation, or a stable data attribute. Resolve extracted relative URLs against the document base, respect robots policies and terms of service, rate-limit requests, and treat extracted content as untrusted input.

Evaluate XPath in Browser Code #

Browsers expose XPath through document.evaluate. Provide an explicit context node and request the result form needed by the caller:

const result = document.evaluate(
  ".//a[normalize-space(.)='Download']/@href",
  container,
  null,
  XPathResult.STRING_TYPE,
  null,
);
const href = result.stringValue;

For many nodes, use an ordered snapshot or an iterator:

const result = document.evaluate(
  ".//tr[td]",
  table,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null,
);
for (let index = 0; index < result.snapshotLength; index += 1) {
  const row = result.snapshotItem(index);
  // Process row while the DOM is stable.
}

An iterator becomes invalid after relevant DOM mutation. A locator API is preferable for authored, testable applications; XPath is valuable when an automation task cannot influence the document structure.

Retrieve Multiple Resources with Curl #

curl accepts multiple URLs in one invocation. Output templates can use #1, #2, and later placeholders for URL glob components:

curl --fail --location --remote-name \
  'https://example.test/images/page-[001-010].jpg'

--fail makes HTTP status failures non-successful, --location follows HTTP redirects, and --remote-name uses the remote filename. Set an explicit --output template when a server filename is unsuitable:

curl --fail --location \
  'https://example.test/page/[0001-0217].jp2' \
  --output 'page-#1.jp2'

Use --proxy socks5h://localhost:1080 when a SOCKS proxy must resolve DNS through the proxy. The h matters: socks5 resolves the target locally, while socks5h sends the hostname to the proxy.

Do not bypass a certificate-name mismatch with --insecure. A mismatch means the requested hostname is absent from the certificate’s subject alternative names. Correct the URL or server configuration, then retry. curl -v exposes the TLS and HTTP exchange for diagnosis; remove authorization headers and cookies before sharing its output.

Process Media with FFmpeg #

FFmpeg separates container formats, streams, codecs, and filters. A file can contain video, audio, subtitle, data, and attachment streams; a filename extension does not identify the codec configuration.

Inspect an input before choosing a command:

ffprobe -hide_banner -show_format -show_streams input.mkv

Copy compatible streams without re-encoding:

ffmpeg -i input.mkv -map 0:v:0 -map 0:a? -c copy output.mkv

-map selects streams explicitly. The trailing ? makes an optional audio stream non-fatal. Stream copy preserves encoded packets, so it cannot change resolution, frame rate, audio sampling, or codec.

Re-encode when a transformation or compatibility conversion is required:

ffmpeg -i input.mov \
  -vf 'scale=1920:-2:flags=lanczos,format=yuv420p' \
  -c:v libx264 -crf 20 -preset medium \
  -c:a aac -b:a 192k \
  output.mp4

-2 asks the scale filter to choose an even dimension, which many subsampled-pixel formats require. -crf controls quality for constant-quality encoders; lower values request higher quality and larger output. Codec options have encoder-specific semantics, so record the command and encoder version in reproducible media pipelines.

Filter graphs combine audio and video operations:

ffmpeg -i video.mp4 -i narration.wav \
  -filter_complex '[0:a][1:a]amix=inputs=2:duration=first[a]' \
  -map 0:v:0 -map '[a]' -c:v copy -c:a aac mixed.mp4

Place input options before the -i they affect and output options after the input they affect. Preserve timestamps with intent: -ss before -i seeks quickly to a nearby keyframe, while -ss after -i decodes to an accurate position. Test seek behavior on representative inputs.

Address Blocks on Magnetic Disks #

A sector is an addressable device unit. A filesystem block is a filesystem allocation or I/O unit and can span one or more sectors. Modern drives expose 512-byte logical sectors, 4096-byte logical sectors, or 512-byte emulation on 4 KiB physical sectors.

Cylinder-head-sector (CHS) addressing modeled a disk’s geometry:

• A cylinder selected one radius across all platter surfaces.
• A head selected a surface, giving a track at that radius.
• A sector selected a device block on the track.

The legacy BIOS CHS interface encoded up to 1024 cylinders, 255 heads, and 63 sectors per track. At 512 bytes per sector, that limit is 8,422,686,720 bytes (about 7.84 GiB). Drives later reported translated geometries, so CHS values ceased to describe physical platters. Zone-bit recording further ensures outer tracks contain more physical sectors than inner tracks.

Logical block addressing (LBA) replaces geometry with a zero-based linear sector number. Firmware translates an LBA into physical locations internally. Tools should display sectors, bytes, and alignment separately because a partitioning convention based on old CHS cylinders does not express modern device geometry.

Choose MBR or GPT Partitioning #

The Master Boot Record occupies logical block 0. Its final two bytes are 0x55AA; the preceding partition table holds four 16-byte primary entries. An entry contains a status byte, legacy CHS fields, a type byte, a starting LBA, and a sector count.

MBR’s four-entry limit led to the extended-partition convention. One primary entry points to an Extended Boot Record chain, and each EBR describes one logical partition plus a link to the next EBR. The chain is a historical layout convention, not a filesystem feature. Bootstrap code in the MBR chooses or transfers control to a partition’s volume boot record; it does not define the filesystem.

GUID Partition Table (GPT) uses LBA and removes the four-partition limit:

A GPT header records the usable LBA range, disk GUID, partition-entry layout, and CRC32 checksums. Each partition entry contains a type GUID, unique GUID, first and last LBAs, attribute flags, and a UTF-16LE name. Validate both the header and entry-array checksums before attempting repair. The backup GPT at the end of the disk supports recovery only when it agrees with known device size and valid checksums.

Align partitions to a 1 MiB boundary unless a platform specifies another requirement. That alignment works with common physical-sector and RAID-stripe layouts without depending on obsolete CHS boundaries.

Read FAT Volume Structures #

FAT12, FAT16, and FAT32 store a per-cluster allocation table. The name denotes the width of a FAT entry, not the sector size. A volume’s boot sector contains the BIOS Parameter Block (BPB), including bytes per sector, sectors per cluster, reserved-sector count, FAT count, root-directory information, and total-sector count.

The root directory is fixed after the FATs in FAT12 and FAT16. FAT32 stores the root as an ordinary cluster chain, with its initial cluster recorded in the extended BPB. Directory entries map a short 8.3 name, attributes, first cluster, size, and timestamps to a file. Long File Name entries precede the short entry and store UTF-16 name fragments with a checksum.

File data is a cluster chain:

1. Locate the first cluster from the directory entry.
2. Read the cluster’s data.
3. Read that cluster’s FAT entry.
4. Continue until an end-of-chain marker appears.

Free, reserved, bad, and end-of-chain values are format-specific. Validate the BPB-derived cluster count before deciding whether a volume is FAT12, FAT16, or FAT32; the textual label in a boot sector is not authoritative. FAT has no journal, so an interrupted metadata update can leave lost chains or cross-linked clusters for a repair utility to reconcile.

Place ATA in the Storage Stack #

IDE was a PC integration name for an ATA disk interface. Parallel ATA (PATA) carried ATA commands over a 40- or 80-conductor ribbon cable and exposed primary and secondary channels with device-select conventions. Serial ATA (SATA) retained the ATA command model while replacing the physical transport with point-to-point serial links.

ATA identifies a drive and its capacity through the IDENTIFY DEVICE command. Commands transfer logical sectors through task-file registers or DMA engines; the controller, cable, host adapter, and device determine transfer mode and error handling. ATAPI used packet commands over an ATA transport for devices such as optical drives, while SCSI command sets dominate many other storage transports.

Avoid interpreting a drive’s reported logical CHS fields as physical geometry. Use LBA capacity and logical/physical sector-size information reported by the device and operating system.

Store and Read Optical Media #

Optical media records a spiral track. A focused laser and photodiode measure changes in reflected light. For a pressed CD, pits and lands are molded into polycarbonate; the signal changes at transitions because the depth difference creates destructive interference across parts of the reflected spot. A pit is not directly one binary value, and transition timing is encoded with run-length-limited channel coding plus error correction.

CD uses a near-infrared laser and a 1.6 µm track pitch. DVD and Blu-ray use shorter wavelengths and smaller pitches, increasing capacity. Constant linear velocity keeps data rate stable by reducing angular speed as the pickup moves outward; later formats can use other rotation strategies.

Pressed discs are read-only. Recordable media changes a dye or inorganic recording layer, while rewritable media uses a phase-change alloy. CD, DVD, and Blu-ray specifications cover physical format, modulation, error correction, and logical formats; a file image format alone does not specify a disc’s physical recording method.

ISO 9660 is a read-only optical-disc filesystem derived from the High Sierra format. It emphasizes sequential layout and cross-platform interchange. Its original naming and directory constraints led to extensions such as Rock Ridge for POSIX metadata and Joliet for longer Unicode-oriented names. El Torito adds boot records for bootable optical media.

UDF supersedes ISO 9660 for many DVD and Blu-ray uses. Its descriptor-based layout supports incremental updates and can represent rewritable and write-once media more naturally. An .iso file is a sector image; it can contain ISO 9660, UDF, or a hybrid layout.

Useful inspection tools include isosize from util-linux and libcdio utilities such as iso-info and iso-read. Mount or inspect untrusted images with the same caution applied to untrusted archives, because filenames and metadata can be hostile input.