Step 6: MD5 Checksum Generation - 14 tests passing

DEMO_step6.md

@@ -0,0 +1,143 @@
+# Step 6: MD5 Checksum Generation - DEMO
+
+## Overview
+This step implements MD5 checksum generation and verification for HathiTrust package validation.
+
+## Key Components
+
+### ChecksumGenerator Class
+Located in `checksum_generator.py`, provides:
+- `compute_md5(file_path)` - Calculate MD5 hash for individual files
+- `generate_checksums(package_directory)` - Create checksum.md5 for all package files
+- `verify_checksums(checksum_file)` - Validate checksums against actual files
+
+### HathiTrust Compliance
+- **Format**: `<hash>  <filename>` (two spaces between hash and filename)
+- **Exclusion**: checksum.md5 does not include itself
+- **Sorting**: Files listed in alphabetical order
+- **Coverage**: All package files (TIFF, TXT, HTML, meta.yml)
+
+## Usage Example
+
+### Generate Checksums
+```python
+from checksum_generator import ChecksumGenerator
+
+generator = ChecksumGenerator()
+result = generator.generate_checksums('/path/to/package')
+
+print(f"Generated checksums for {result['file_count']} files")
+print(f"Checksum file: {result['checksum_file']}")
+```
+
+### Verify Checksums
+```python
+verify_result = generator.verify_checksums('/path/to/package/checksum.md5')
+
+print(f"Valid: {len(verify_result['valid'])}")
+print(f"Invalid: {len(verify_result['invalid'])}")
+print(f"Missing: {len(verify_result['missing'])}")
+```
+
+## Test Results
+✅ **14 tests passed** (0.05s)
+
+### Test Coverage
+1. ✅ Basic MD5 computation
+2. ✅ MD5 consistency (same file → same hash)
+3. ✅ Error handling (missing files)
+4. ✅ Checksum.md5 file generation
+5. ✅ File format compliance (hash  filename)
+6. ✅ Self-exclusion (checksum.md5 not in itself)
+7. ✅ Sorted order verification
+8. ✅ Validation of valid checksums
+9. ✅ Detection of modified files
+10. ✅ Detection of missing files
+11. ✅ Empty directory error handling
+12. ✅ Nonexistent directory error handling
+13. ✅ Convenience function
+14. ✅ Binary file (TIFF) checksums
+
+
+## Sample checksum.md5 File
+
+```
+00000001.html  a3c1f5e9d4b2c8f7e6d5a4b3c2d1e0f9
+00000001.tif  b2d3e4f5c6a7b8c9d0e1f2a3b4c5d6e7
+00000001.txt  c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9
+00000002.html  d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0
+00000002.tif  e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1
+00000002.txt  f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2
+meta.yml  a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3
+```
+
+## Technical Implementation
+
+### MD5 Computation
+- **Chunk size**: 8KB for memory efficiency
+- **Encoding**: Works with both binary (TIFF) and text files
+- **Output**: Lowercase hexadecimal (32 characters)
+
+### Error Handling
+- `FileNotFoundError` - File doesn't exist
+- `IOError` - File cannot be read
+- `NotADirectoryError` - Invalid package directory
+- `ValueError` - No files found in directory
+
+### Verification Features
+- Detects modified files (checksum mismatch)
+- Identifies missing files (in checksum.md5 but not found)
+- Confirms valid files (checksums match)
+- Returns detailed results for reporting
+
+## Integration with Pipeline
+
+### Position in Workflow
+```
+Step 5: YAML Generation → Step 6: Checksum Generation → Step 7: Package Assembly
+```
+
+### When to Generate Checksums
+- **After** all package files are finalized (TIFF, TXT, HTML, meta.yml)
+- **Before** creating ZIP archive
+- **Last step** before packaging to ensure file integrity
+
+### Checksum Verification Use Cases
+1. **Pre-transfer**: Verify package integrity before upload
+2. **Post-transfer**: Validate files after network transfer
+3. **Archive validation**: Periodic checks on stored packages
+4. **Error recovery**: Identify corrupted files in batch processing
+
+## Next Steps
+
+### Step 7: Package Assembly
+Create `package_assembler.py` to:
+- Organize all files into flat directory structure
+- Copy/move TIFF, TXT, HTML, meta.yml into package directory
+- Validate file naming conventions
+- Prepare for ZIP creation
+
+### Integration Points
+```python
+# Step 7 will use checksum_generator like this:
+from checksum_generator import generate_package_checksums
+
+# After assembling package files...
+checksum_file = generate_package_checksums(package_dir)
+print(f"Package ready for ZIP: {checksum_file}")
+```
+
+## Dependencies Updated
+Added to `requirements.txt`:
+```
+pytest>=8.0.0  # Testing framework
+```
+
+## Files Created
+- `checksum_generator.py` - Main implementation (131 lines)
+- `test_checksum_generator.py` - Test suite (149 lines)
+- `DEMO_step6.md` - Documentation (this file)
+
+---
+
+**Status**: ✅ Step 6 Complete | 14/14 Tests Passing | Ready for Step 7

checksum_generator.py

@@ -0,0 +1,167 @@
+"""
+HathiTrust Package Automation - Step 6: MD5 Checksum Generation
+Computes MD5 hashes for all package files and creates checksum.md5
+"""
+
+import hashlib
+import os
+from pathlib import Path
+from typing import Dict, List
+
+
+class ChecksumGenerator:
+    """Generates MD5 checksums for HathiTrust package files"""
+
+    def __init__(self):
+        self.chunk_size = 8192  # 8KB chunks for efficient memory usage
+
+    def compute_md5(self, file_path: str) -> str:
+        """
+        Calculate MD5 hash of a file.
+        
+        Args:
+            file_path: Path to file to hash
+            
+        Returns:
+            MD5 hash as lowercase hexadecimal string
+            
+        Raises:
+            FileNotFoundError: If file doesn't exist
+            IOError: If file cannot be read
+        """
+        if not os.path.exists(file_path):
+            raise FileNotFoundError(f"File not found: {file_path}")
+
+        md5_hasher = hashlib.md5()
+
+        try:
+            with open(file_path, 'rb') as f:
+                for chunk in iter(lambda: f.read(self.chunk_size), b''):
+                    md5_hasher.update(chunk)
+        except IOError as e:
+            raise IOError(f"Error reading file {file_path}: {e}")
+
+        return md5_hasher.hexdigest()
+
+    def generate_checksums(self, package_directory: str, output_file: str = "checksum.md5") -> Dict:
+        """
+        Generate checksum.md5 file for all files in package directory.
+        
+        Args:
+            package_directory: Path to directory containing package files
+            output_file: Name of checksum file (default: checksum.md5)
+            
+        Returns:
+            Dictionary with:
+                - checksums: List of (hash, filename) tuples
+                - checksum_file: Path to generated checksum.md5
+                - file_count: Number of files processed
+                
+        Raises:
+            NotADirectoryError: If package_directory doesn't exist or isn't a directory
+        """
+        package_path = Path(package_directory)
+
+        if not package_path.exists():
+            raise NotADirectoryError(f"Directory not found: {package_directory}")
+
+        if not package_path.is_dir():
+            raise NotADirectoryError(f"Not a directory: {package_directory}")
+
+        checksums = []
+
+        # Get all files in directory (excluding checksum.md5 itself)
+        for file_path in sorted(package_path.iterdir()):
+            if file_path.is_file() and file_path.name != output_file:
+                md5_hash = self.compute_md5(str(file_path))
+                filename = file_path.name
+                checksums.append((md5_hash, filename))
+
+        if not checksums:
+            raise ValueError(f"No files found in {package_directory}")
+
+        # Write checksum file (format: <hash>  <filename>)
+        checksum_path = package_path / output_file
+        with open(checksum_path, 'w', encoding='utf-8') as f:
+            for md5_hash, filename in checksums:
+                f.write(f"{md5_hash}  {filename}\n")  # Two spaces per HathiTrust spec
+
+        return {
+            'checksums': checksums,
+            'checksum_file': str(checksum_path),
+            'file_count': len(checksums)
+        }
+
+    def verify_checksums(self, checksum_file: str) -> Dict:
+        """
+        Verify checksums in a checksum.md5 file.
+        
+        Args:
+            checksum_file: Path to checksum.md5 file
+            
+        Returns:
+            Dictionary with:
+                - valid: List of validated files
+                - invalid: List of (filename, expected_hash, actual_hash) for mismatches
+                - missing: List of files in checksum.md5 but not found
+                - total: Total files checked
+                
+        Raises:
+            FileNotFoundError: If checksum file doesn't exist
+        """
+        checksum_path = Path(checksum_file)
+
+        if not checksum_path.exists():
+            raise FileNotFoundError(f"Checksum file not found: {checksum_file}")
+
+        package_dir = checksum_path.parent
+        valid = []
+        invalid = []
+        missing = []
+
+        with open(checksum_path, 'r', encoding='utf-8') as f:
+            for line in f:
+                line = line.strip()
+                if not line:
+                    continue
+
+                # Parse checksum line: <hash>  <filename>
+                parts = line.split(None, 1)  # Split on whitespace, max 2 parts
+                if len(parts) != 2:
+                    continue
+
+                expected_hash, filename = parts
+                file_path = package_dir / filename
+
+                if not file_path.exists():
+                    missing.append(filename)
+                    continue
+
+                actual_hash = self.compute_md5(str(file_path))
+
+                if actual_hash == expected_hash:
+                    valid.append(filename)
+                else:
+                    invalid.append((filename, expected_hash, actual_hash))
+
+        return {
+            'valid': valid,
+            'invalid': invalid,
+            'missing': missing,
+            'total': len(valid) + len(invalid) + len(missing)
+        }
+
+
+def generate_package_checksums(package_directory: str) -> str:
+    """
+    Convenience function to generate checksums for a package.
+    
+    Args:
+        package_directory: Path to package directory
+        
+    Returns:
+        Path to generated checksum.md5 file
+    """
+    generator = ChecksumGenerator()
+    result = generator.generate_checksums(package_directory)
+    return result['checksum_file']

requirements.txt

@@ -3,3 +3,5 @@ pytesseract>=0.3.10
 PyYAML>=6.0
 Pillow>=10.0.0
 tqdm>=4.65.0
+
+pytest>=8.0.0  # Testing framework