daengdeni/lpj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

✨ feat(slik): implementasi sistem import SLIK dengan optimasi memory & timeout handling

Browse Source 
- Menambahkan `SlikController.php` dengan method CRUD dan import data SLIK, termasuk logging detail & error handling
- Menambahkan `SlikImport.php` dengan Laravel Excel (ToCollection, WithChunkReading, WithBatchInserts, dll.)
- Optimasi memory dengan chunk processing (50 baris/chunk) dan batch insert (50 record/batch)
- Penanganan timeout menggunakan `set_time_limit` & memory limit configurable via config
- Implementasi queue processing untuk file besar (>5MB) dengan progress tracking
- Validasi file upload & data baris, skip header dari baris ke-5, serta rollback jika error
- Garbage collection otomatis setiap 25 baris, unset variabel tidak terpakai, dan logging usage memory
- Error handling komprehensif dengan try-catch, rollback transaksi, hapus file temp, dan logging stack trace
- Semua parameter (batch size, chunk size, memory limit, timeout, GC, queue threshold) configurable via config
- Diuji pada file besar (>50MB), memory stabil, timeout handling berfungsi, rollback aman, dan progress tracking valid
- Catatan: pastikan queue worker berjalan, monitor log progress, sesuaikan config server, dan backup DB sebelum import
This commit is contained in:
Daeng Deni Mardaeni
2025-09-16 11:54:39 +07:00
parent 81159983cf
commit 20833213b1
11 changed files with 2032 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
config/config.php
View File

@@ -2,4 +2,62 @@
return [
'name' => 'Lpj',
'import' => [
'slik' => [
// Memory limit untuk import (dalam MB)
'memory_limit' => env('SLIK_IMPORT_MEMORY_LIMIT', 1024),
// Ukuran chunk untuk processing (jumlah baris per chunk)
'chunk_size' => env('SLIK_IMPORT_CHUNK_SIZE', 50),
// Ukuran batch untuk database insert
'batch_size' => env('SLIK_IMPORT_BATCH_SIZE', 50),
// Timeout untuk import (dalam detik)
'timeout' => env('SLIK_IMPORT_TIMEOUT', 1800), // 30 menit untuk file besar
// Maksimum file size yang diizinkan (dalam MB)
'max_file_size' => env('SLIK_IMPORT_MAX_FILE_SIZE', 50),
// Enable garbage collection untuk optimasi memory
'enable_gc' => env('SLIK_IMPORT_ENABLE_GC', true),
// Enable progress logging
'enable_progress_logging' => env('SLIK_IMPORT_ENABLE_PROGRESS_LOGGING', true),
// Enable detailed error logging
'enable_error_logging' => env('SLIK_IMPORT_ENABLE_ERROR_LOGGING', true),
// XML Scanner settings untuk optimasi memory
'xml_scanner' => [
'timeout' => env('SLIK_XML_SCANNER_TIMEOUT', 1800), // 30 menit
'memory_limit' => env('SLIK_XML_SCANNER_MEMORY_LIMIT', 1024), // 1GB
'chunk_size' => env('SLIK_XML_SCANNER_CHUNK_SIZE', 50), // Lebih kecil untuk XML
],
// Queue processing untuk file besar
'queue' => [
'enabled' => env('SLIK_IMPORT_QUEUE_ENABLED', false),
'connection' => env('SLIK_IMPORT_QUEUE_CONNECTION', 'database'),
'queue_name' => env('SLIK_IMPORT_QUEUE_NAME', 'imports'),
'chunk_size' => env('SLIK_IMPORT_QUEUE_CHUNK_SIZE', 500),
],
// Progress tracking
'progress' => [
'enabled' => env('SLIK_IMPORT_PROGRESS_ENABLED', true),
'update_interval' => env('SLIK_IMPORT_PROGRESS_INTERVAL', 50), // update setiap 50 baris
'cache_key' => 'slik_import_progress',
'cache_ttl' => 3600, // 1 jam
],
],
// General import settings
'general' => [
'default_memory_limit' => env('IMPORT_DEFAULT_MEMORY_LIMIT', 128),
'max_execution_time' => env('IMPORT_MAX_EXECUTION_TIME', 300000),
'temp_directory' => env('IMPORT_TEMP_DIRECTORY', storage_path('app/temp')),
'cleanup_temp_files' => env('IMPORT_CLEANUP_TEMP_FILES', true),
],
],
];