feat(webstatement): implement batch processing for data jobs

- Tambahkan pemrosesan data secara bertahap (chunking) dengan konstanta `CHUNK_SIZE` untuk mengurangi penggunaan memori dan menangani data dalam jumlah besar. - Perbarui `ProcessArrangementDataJob`: - Tambahkan properti `arrangementBatch` untuk menyimpan batch data sementara. - Implementasikan metode `addToBatch` untuk menambahkan data ke batch. - Implementasikan metode `saveBatch` untuk menyimpan data batch ke database menggunakan metode bulk `TempArrangement::upsert`. - Tambahkan logging untuk melacak progress pemrosesan data per chunk. - Reset batch setelah penyimpanan atau ketika terjadi error untuk menghindari pemrosesan ulang data yang gagal. - Perbarui `ProcessBillDetailDataJob`: - Tambahkan properti `billDetailBatch` untuk menyimpan batch data sementara. - Implementasikan metode `addToBatch` untuk menambahkan data ke batch. - Implementasikan metode `saveBatch` untuk menyimpan data batch ke database menggunakan metode bulk `TempBillDetail::upsert`. - Tambahkan logging untuk melacak progress pemrosesan data per chunk. - Reset batch setelah penyimpanan atau ketika terjadi error untuk menghindari pemrosesan ulang data yang gagal. - Perbaiki penghitungan error count dengan menambahkannya saat terjadi error pada pemrosesan batch. - Tambahkan timestamp (`created_at` dan `updated_at`) pada setiap record dalam batch sebelum disimpan ke database. - Lakukan cleanup batch secara otomatis setelah pemrosesan selesai.
feat(webstatement): implement batch processing in job data handling
2025-05-28 09:29:46 +07:00 · 2025-05-28 09:29:18 +07:00 · 2025-05-28 09:28:12 +07:00 · 2025-05-28 09:27:49 +07:00
8 changed files with 438 additions and 49 deletions
--- a/app/Jobs/ProcessArrangementDataJob.php
+++ b/app/Jobs/ProcessArrangementDataJob.php
@@ -20,10 +20,12 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.AA.ARRANGEMENT.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage

        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $arrangementBatch = [];

        /**
         * Create a new job instance.
@@ -61,6 +63,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->arrangementBatch = [];
        }

        private function processPeriod()
@@ -111,10 +114,23 @@

            $headers  = (new TempArrangement())->getFillable();
            $rowCount = 0;
+            $chunkCount = 0;

            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($row, $headers, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->arrangementBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->arrangementBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -127,22 +143,29 @@
            if (count($headers) !== count($row)) {
                Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " .
                    count($headers) . ", Got: " . count($row));
+                $this->errorCount++;
                return;
            }

            $data = array_combine($headers, $row);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if ($data['arrangement_id'] !== 'arrangement_id') {
-                    TempArrangement::updateOrCreate(
-                        ['arrangement_id' => $data['arrangement_id']], // key to find existing record
-                        $data                                          // data to update or create
-                    );
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to batch
+                    $this->arrangementBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -151,6 +174,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->arrangementBatch)) {
+                    // Bulk insert/update arrangements
+                    TempArrangement::upsert(
+                        $this->arrangementBatch,
+                        ['arrangement_id'], // Unique key
+                        array_diff((new TempArrangement())->getFillable(), ['arrangement_id']) // Update columns
+                    );
+
+                    // Reset batch after processing
+                    $this->arrangementBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->arrangementBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->arrangementBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessAtmTransactionJob.php
+++ b/app/Jobs/ProcessAtmTransactionJob.php
@@ -20,6 +20,7 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.ATM.TRANSACTION.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage
        private const HEADER_MAP         = [
            'id'               => 'transaction_id',
            'card_acc_id'      => 'card_acc_id',
@@ -43,6 +44,7 @@
        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $atmTransactionBatch = [];

        /**
         * Create a new job instance.
@@ -80,6 +82,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->atmTransactionBatch = [];
        }

        private function processPeriod()
@@ -136,9 +139,23 @@
            }

            $rowCount = 0;
+            $chunkCount = 0;
+
            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($headerRow, $row, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->atmTransactionBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->atmTransactionBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -151,15 +168,16 @@
            if (count($headerRow) !== count($row)) {
                Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " .
                    count($headerRow) . ", Got: " . count($row));
+                $this->errorCount++;
                return;
            }

            // Combine the header row with the data row
            $rawData = array_combine($headerRow, $row);
-            $this->mapAndSaveRecord($rawData, $rowCount, $filePath);
+            $this->mapAndAddToBatch($rawData, $rowCount, $filePath);
        }

-        private function mapAndSaveRecord(array $rawData, int $rowCount, string $filePath)
+        private function mapAndAddToBatch(array $rawData, int $rowCount, string $filePath)
        : void
        {
            // Map the raw data to our model fields
@@ -173,24 +191,53 @@
                return;
            }

-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
+                // Add timestamp fields
+                $now = now();
+                $data['created_at'] = $now;
+                $data['updated_at'] = $now;

-                // Create or update the record
-                AtmTransaction::updateOrCreate(
-                    ['transaction_id' => $data['transaction_id']],
-                    $data
-                );
-
+                // Add to batch
+                $this->atmTransactionBatch[] = $data;
                $this->processedCount++;
            } catch (Exception $e) {
                $this->errorCount++;
-                Log::error("Error processing row $rowCount in $filePath: " . $e->getMessage());
+                Log::error("Error processing ATM Transaction at row $rowCount in $filePath: " . $e->getMessage());
+            }
+        }
+
+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->atmTransactionBatch)) {
+                    // Bulk insert/update ATM transactions
+                    AtmTransaction::upsert(
+                        $this->atmTransactionBatch,
+                        ['transaction_id'], // Unique key
+                        array_values(array_diff(self::HEADER_MAP, ['id'])) // Update columns (all except transaction_id)
+                    );
+
+                    // Reset batch after processing
+                    $this->atmTransactionBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->atmTransactionBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->atmTransactionBatch = [];
            }
        }

--- a/app/Jobs/ProcessBillDetailDataJob.php
+++ b/app/Jobs/ProcessBillDetailDataJob.php
@@ -20,10 +20,12 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.AA.BILL.DETAILS.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage

        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $billDetailBatch = [];

        /**
         * Create a new job instance.
@@ -61,6 +63,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->billDetailBatch = [];
        }

        private function processPeriod()
@@ -111,10 +114,23 @@

            $headers  = (new TempBillDetail())->getFillable();
            $rowCount = 0;
+            $chunkCount = 0;

            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($row, $headers, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->billDetailBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->billDetailBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -127,22 +143,29 @@
            if (count($headers) !== count($row)) {
                Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " .
                    count($headers) . ", Got: " . count($row));
+                $this->errorCount++;
                return;
            }

            $data = array_combine($headers, $row);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if (isset($data['_id']) && $data['_id'] !== '_id') {
-                    TempBillDetail::updateOrCreate(
-                        ['_id' => $data['_id']],
-                        $data
-                    );
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to batch
+                    $this->billDetailBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -151,6 +174,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->billDetailBatch)) {
+                    // Bulk insert/update bill details
+                    TempBillDetail::upsert(
+                        $this->billDetailBatch,
+                        ['_id'], // Unique key
+                        array_diff((new TempBillDetail())->getFillable(), ['_id']) // Update columns
+                    );
+
+                    // Reset batch after processing
+                    $this->billDetailBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->billDetailBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->billDetailBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessCustomerDataJob.php
+++ b/app/Jobs/ProcessCustomerDataJob.php
@@ -20,10 +20,12 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.CUSTOMER.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage

        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $customerBatch  = [];

        /**
         * Create a new job instance.
@@ -61,6 +63,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->customerBatch  = [];
        }

        private function processPeriod()
@@ -111,10 +114,23 @@

            $headers  = (new Customer())->getFillable();
            $rowCount = 0;
+            $chunkCount = 0;

            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($row, $headers, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->customerBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->customerBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -131,17 +147,24 @@
            }

            $data = array_combine($headers, $row);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if (isset($data['customer_code']) && $data['customer_code'] !== 'customer_code') {
-                    $customer = Customer::firstOrNew(['customer_code' => $data['customer_code']]);
-                    $customer->fill($data);
-                    $customer->save();
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to customer batch
+                    $this->customerBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -150,6 +173,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->customerBatch)) {
+                    // Bulk insert/update customers
+                    Customer::upsert(
+                        $this->customerBatch,
+                        ['customer_code'], // Unique key
+                        array_diff((new Customer())->getFillable(), ['customer_code']) // Update columns
+                    );
+
+                    // Reset customer batch after processing
+                    $this->customerBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->customerBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->customerBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessDataCaptureDataJob.php
+++ b/app/Jobs/ProcessDataCaptureDataJob.php
@@ -20,6 +20,7 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.DATA.CAPTURE.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage
        private const CSV_HEADERS        = [
            'id',
            'account_number',
@@ -66,6 +67,7 @@
        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $captureBatch   = [];

        /**
         * Create a new job instance.
@@ -103,6 +105,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->captureBatch   = [];
        }

        private function processPeriod()
@@ -152,6 +155,8 @@
            }

            $rowCount = 0;
+            $chunkCount = 0;
+
            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;

@@ -161,6 +166,18 @@
                }

                $this->processRow($row, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->captureBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->captureBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -178,7 +195,7 @@

            $data = array_combine(self::CSV_HEADERS, $row);
            $this->formatDates($data);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

        private function formatDates(array &$data)
@@ -207,15 +224,21 @@
            }
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if (!empty($data['id'])) {
-                    DataCapture::updateOrCreate(
-                        ['id' => $data['id']],
-                        $data
-                    );
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to capture batch
+                    $this->captureBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -224,6 +247,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->captureBatch)) {
+                    // Bulk insert/update data captures
+                    DataCapture::upsert(
+                        $this->captureBatch,
+                        ['id'], // Unique key
+                        array_diff(self::CSV_HEADERS, ['id']) // Update columns
+                    );
+
+                    // Reset capture batch after processing
+                    $this->captureBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->captureBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->captureBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessFundsTransferDataJob.php
+++ b/app/Jobs/ProcessFundsTransferDataJob.php
@@ -20,10 +20,12 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.FUNDS.TRANSFER.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage

        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $transferBatch  = [];

        /**
         * Create a new job instance.
@@ -61,6 +63,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->transferBatch  = [];
        }

        private function processPeriod()
@@ -111,10 +114,23 @@

            $headers  = (new TempFundsTransfer())->getFillable();
            $rowCount = 0;
+            $chunkCount = 0;

            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($row, $headers, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->transferBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->transferBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -136,18 +152,24 @@
            }

            $data = array_combine($headers, $row);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if (isset($data['_id']) && $data['_id'] !== '_id') {
-                    TempFundsTransfer::updateOrCreate(
-                        ['_id' => $data['_id']],
-                        $data
-                    );
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to transfer batch
+                    $this->transferBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -156,6 +178,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->transferBatch)) {
+                    // Bulk insert/update funds transfers
+                    TempFundsTransfer::upsert(
+                        $this->transferBatch,
+                        ['_id'], // Unique key
+                        array_diff((new TempFundsTransfer())->getFillable(), ['_id']) // Update columns
+                    );
+
+                    // Reset transfer batch after processing
+                    $this->transferBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->transferBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->transferBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessStmtEntryDataJob.php
+++ b/app/Jobs/ProcessStmtEntryDataJob.php
@@ -20,10 +20,12 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.STMT.ENTRY.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage

        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $entryBatch     = [];

        /**
         * Create a new job instance.
@@ -61,6 +63,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->entryBatch     = [];
        }

        private function processPeriod()
@@ -111,10 +114,23 @@

            $headers  = (new StmtEntry())->getFillable();
            $rowCount = 0;
+            $chunkCount = 0;

            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($row, $headers, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->entryBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->entryBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -132,7 +148,7 @@

            $data = array_combine($headers, $row);
            $this->cleanTransReference($data);
-            $this->saveRecord($data, $rowCount, $filePath);
+            $this->addToBatch($data, $rowCount, $filePath);
        }

        private function cleanTransReference(array &$data)
@@ -144,15 +160,21 @@
            }
        }

-        private function saveRecord(array $data, int $rowCount, string $filePath)
+        /**
+         * Add record to batch instead of saving immediately
+         */
+        private function addToBatch(array $data, int $rowCount, string $filePath)
        : void
        {
            try {
                if (isset($data['stmt_entry_id']) && $data['stmt_entry_id'] !== 'stmt_entry_id') {
-                    StmtEntry::updateOrCreate(
-                        ['stmt_entry_id' => $data['stmt_entry_id']],
-                        $data
-                    );
+                    // Add timestamp fields
+                    $now = now();
+                    $data['created_at'] = $now;
+                    $data['updated_at'] = $now;
+
+                    // Add to entry batch
+                    $this->entryBatch[] = $data;
                    $this->processedCount++;
                }
            } catch (Exception $e) {
@@ -161,6 +183,32 @@
            }
        }

+        /**
+         * Save batched records to the database
+         */
+        private function saveBatch()
+        : void
+        {
+            try {
+                if (!empty($this->entryBatch)) {
+                    // Bulk insert/update statement entries
+                    StmtEntry::upsert(
+                        $this->entryBatch,
+                        ['stmt_entry_id'], // Unique key
+                        array_diff((new StmtEntry())->getFillable(), ['stmt_entry_id']) // Update columns
+                    );
+
+                    // Reset entry batch after processing
+                    $this->entryBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->entryBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->entryBatch = [];
+            }
+        }
+
        private function cleanup(string $tempFilePath)
        : void
        {
--- a/app/Jobs/ProcessTellerDataJob.php
+++ b/app/Jobs/ProcessTellerDataJob.php
@@ -20,6 +20,7 @@
        private const MAX_EXECUTION_TIME = 86400; // 24 hours in seconds
        private const FILENAME           = 'ST.TELLER.csv';
        private const DISK_NAME          = 'sftpStatement';
+        private const CHUNK_SIZE         = 1000; // Process data in chunks to reduce memory usage
        private const HEADER_MAP         = [
            'id'                => 'id_teller',
            'account_1'         => 'account_1',
@@ -129,6 +130,7 @@
        private string $period = '';
        private int    $processedCount = 0;
        private int    $errorCount     = 0;
+        private array  $tellerBatch    = [];

        /**
         * Create a new job instance.
@@ -166,6 +168,7 @@
            set_time_limit(self::MAX_EXECUTION_TIME);
            $this->processedCount = 0;
            $this->errorCount     = 0;
+            $this->tellerBatch    = [];
        }

        private function processPeriod()
@@ -222,9 +225,23 @@
            }

            $rowCount = 0;
+            $chunkCount = 0;
+
            while (($row = fgetcsv($handle, 0, self::CSV_DELIMITER)) !== false) {
                $rowCount++;
                $this->processRow($headerRow, $row, $rowCount, $filePath);
+
+                // Process in chunks to avoid memory issues
+                if (count($this->tellerBatch) >= self::CHUNK_SIZE) {
+                    $this->saveBatch();
+                    $chunkCount++;
+                    Log::info("Processed chunk $chunkCount ({$this->processedCount} records so far)");
+                }
+            }
+
+            // Process any remaining records
+            if (!empty($this->tellerBatch)) {
+                $this->saveBatch();
            }

            fclose($handle);
@@ -234,6 +251,14 @@
        private function processRow(array $headerRow, array $row, int $rowCount, string $filePath)
        : void
        {
+            // Skip if row doesn't have enough columns
+            if (count($headerRow) !== count($row)) {
+                Log::warning("Row $rowCount in $filePath has incorrect column count. Expected: " .
+                    count($headerRow) . ", Got: " . count($row));
+                $this->errorCount++;
+                return;
+            }
+
            // Combine the header row with the data row
            $rawData = array_combine($headerRow, $row);

@@ -249,10 +274,13 @@
            }

            try {
-                $teller = Teller::firstOrNew(['id_teller' => $data['id_teller']]);
-                $teller->fill($data);
-                $teller->save();
+                // Add timestamps
+                $now = now();
+                $data['created_at'] = $now;
+                $data['updated_at'] = $now;

+                // Add to batch for bulk processing
+                $this->tellerBatch[] = $data;
                $this->processedCount++;
            } catch (Exception $e) {
                Log::error("Error processing Teller at row $rowCount in $filePath: " . $e->getMessage());
@@ -260,6 +288,28 @@
            }
        }

+        private function saveBatch(): void
+        {
+            try {
+                if (!empty($this->tellerBatch)) {
+                    // Bulk insert/update teller records
+                    Teller::upsert(
+                        $this->tellerBatch,
+                        ['id_teller'], // Unique key
+                        array_diff(array_values(self::HEADER_MAP), ['id_teller']) // Update columns
+                    );
+
+                    // Reset batch after processing
+                    $this->tellerBatch = [];
+                }
+            } catch (Exception $e) {
+                Log::error("Error in saveBatch: " . $e->getMessage());
+                $this->errorCount += count($this->tellerBatch);
+                // Reset batch even if there's an error to prevent reprocessing the same failed records
+                $this->tellerBatch = [];
+            }
+        }
+
        private function logJobCompletion()
        : void
        {
Author	SHA1	Message	Date
daengdeni	1b3e0ed30d	feat(webstatement): implement batch processing for data jobs - Tambahkan pemrosesan data secara bertahap (chunking) dengan konstanta `CHUNK_SIZE` untuk mengurangi penggunaan memori dan menangani data dalam jumlah besar. - Perbarui `ProcessArrangementDataJob`: - Tambahkan properti `arrangementBatch` untuk menyimpan batch data sementara. - Implementasikan metode `addToBatch` untuk menambahkan data ke batch. - Implementasikan metode `saveBatch` untuk menyimpan data batch ke database menggunakan metode bulk `TempArrangement::upsert`. - Tambahkan logging untuk melacak progress pemrosesan data per chunk. - Reset batch setelah penyimpanan atau ketika terjadi error untuk menghindari pemrosesan ulang data yang gagal. - Perbarui `ProcessBillDetailDataJob`: - Tambahkan properti `billDetailBatch` untuk menyimpan batch data sementara. - Implementasikan metode `addToBatch` untuk menambahkan data ke batch. - Implementasikan metode `saveBatch` untuk menyimpan data batch ke database menggunakan metode bulk `TempBillDetail::upsert`. - Tambahkan logging untuk melacak progress pemrosesan data per chunk. - Reset batch setelah penyimpanan atau ketika terjadi error untuk menghindari pemrosesan ulang data yang gagal. - Perbaiki penghitungan error count dengan menambahkannya saat terjadi error pada pemrosesan batch. - Tambahkan timestamp (`created_at` dan `updated_at`) pada setiap record dalam batch sebelum disimpan ke database. - Lakukan cleanup batch secara otomatis setelah pemrosesan selesai.	2025-05-28 09:29:46 +07:00
daengdeni	0b607f86cb	feat(webstatement): implement batch processing in job data handling - Menambahkan konstanta `CHUNK_SIZE` dengan nilai 1000 untuk memungkinkan pemrosesan data dalam bentuk batch guna mengurangi penggunaan memori. - Memperkenalkan atribut batch baru (`atmTransactionBatch`, `captureBatch`, `transferBatch`, dan `tellerBatch`) untuk menyimpan data sementara sebelum disimpan ke database. - Mengganti metode penyimpanan langsung dengan menambahkan data ke batch melalui fungsi baru `addToBatch()`. - Menambahkan fungsi `saveBatch()` untuk melakukan operasi penyimpanan data dalam jumlah besar (bulk) menggunakan metode `upsert`. - Memastikan batch akan di-reset setelah penyimpanan, termasuk dalam kasus kegagalan, untuk mencegah reprocessing catatan yang gagal. - Menambahkan logging tambahan untuk mencatat jumlah batch (chunk) yang telah selesai diproses, memberikan wawasan lebih dalam proses. - Menambahkan validasi jumlah kolom pada setiap baris data, logging peringatan jika data tidak sesuai, dan mencatat jumlah kesalahan (`errorCount`). - Secara otomatis menambahkan atribut timestamp (`created_at` dan `updated_at`) pada setiap data sebelum dimasukkan ke dalam batch untuk pelacakan waktu. - Memodifikasi log error untuk menangani kesalahan pada level yang lebih spesifik, seperti pada baris tertentu dalam file yang diproses. - Mengoptimalkan pemrosesan data pada job: - `ProcessAtmTransactionJob` untuk data transaksi ATM. - `ProcessDataCaptureDataJob` untuk data capture. - `ProcessFundsTransferDataJob` untuk data transfer dana. - `ProcessTellerDataJob` untuk data teller. - Meningkatkan efisiensi dan skalabilitas dengan menyisipkan data secara bulk, mengurangi overhead database, dan menghindari pengolahan data secara satu per satu.	2025-05-28 09:29:18 +07:00
daengdeni	30662b97d5	feat(webstatement): optimize data processing with batching strategy - Menambahkan konstanta `CHUNK_SIZE` untuk membatasi ukuran batch selama proses data. - Memperkenalkan properti `$entryBatch` untuk menyimpan batch data sementara sebelum disimpan ke database. - Mengubah metode `saveRecord` menjadi `addToBatch` untuk menambahkan record ke batch alih-alih langsung menyimpannya. - Menambahkan metode `saveBatch` untuk menyimpan batch data ke database secara efisien menggunakan `upsert`. - Menangani chunking dalam proses CSV untuk menghindari masalah memori dengan memproses data dalam batch. - Menambahkan logging pada setiap batch yang selesai diproses untuk melacak kemajuan pemrosesan. - Memastikan batch yang tersisa diproses setelah selesai membaca file CSV. - Menambahkan penanganan error saat menyimpan batch untuk mencegah reprocessing pada batch yang gagal memproses. - Otomatis menambahkan timestamp (`created_at` dan `updated_at`) pada setiap record sebelum dimasukkan ke batch.	2025-05-28 09:28:12 +07:00
daengdeni	cbfe2c4aa9	feat(webstatement): optimize customer data processing with batch handling - Tambahkan konstanta `CHUNK_SIZE` untuk memproses data dalam ukuran batch guna mengurangi penggunaan memori. - Perkenalkan properti baru `customerBatch` untuk menyimpan data sementara sebelum disimpan ke database. - Ubah metode `processRow` agar menambahkan data ke batch menggunakan metode baru `addToBatch` daripada langsung menyimpannya. - Tambahkan metode `saveBatch` untuk melakukan penyimpanan batch secara bulk menggunakan `Customer::upsert` dengan pengelolaan kolom unik dan kolom yang perlu diperbarui. - Tambahkan log untuk setiap chunk yang telah berhasil diproses, membantu memonitor progres saat pemrosesan file CSV dengan ukuran besar. - Pastikan sisa data yang belum diproses di akhir loop juga disimpan dengan memanggil `saveBatch`. - Tangani kegagalan penyimpanan batch dengan log error dan reset batch untuk menghindari re-pemrosesan data yang gagal. - Optimalkan performa dengan menambahkan timestamp (`created_at` dan `updated_at`) secara langsung saat menambahkan data ke batch.	2025-05-28 09:27:49 +07:00