The Task

All AI tools were given the same input to evaluate their capabilities in scripting for database management systems. The task was clearly defined as follows:

Prompt: “Write a FileMaker script that generates 1000 records in a table named myTable within a field called MyField. Upon completion, the script should display a dialog indicating success if no errors occurred, or display an error message if any issues were encountered.”

This prompt tests each AI’s ability to handle loop structures, error handling, and user feedback within the FileMaker scripting environment—a popular database management system known for its robustness and ease of use in creating customized database solutions.

Overview of AI Responses

Each AI platform, including ChatGPT (GPT-4), GPT-3.5, Meta.ai, Mistral.ai, Pi.ai, Perplexity.ai, Copilot Free, Blackbox.ai, Gemini Free, Gemini Pro, and Claude 3, received the same task. The objective was to assess how well these tools could autonomously generate a script that not only performs the task effectively but also adheres to best practices in error management and user interaction within the FileMaker environment.

Analysis and Insights

The responses varied significantly, with some AIs demonstrating a strong grasp of FileMaker scripting syntax and logic, while others showed fundamental misunderstandings or errors in script construction. This analysis highlights the importance of accuracy in AI-generated scripts, especially in scenarios where efficiency and error handling can dramatically impact the usability and functionality of the database application.

For a detailed breakdown of each AI’s script and the revised quality scores based on their performance, refer to the chart and evaluations provided in the subsequent sections. This comprehensive review serves as a critical resource for developers and database administrators who rely on AI tools for generating automation scripts in professional settings.

1. ChatGPT-4 (Paid Version)

Script:

Set Error Capture [On]
Set Variable [$error; Value:""]
Loop
    Exit Loop If [$i = 1000]
    New Record/Request
    Set Field [myTable::MyField; "default value"]
    Set Variable [$error; Value:Get(LastError)]
    If [$error ≠ 0]
        Exit Loop If [True]
    End If
    Set Variable [$i; Value:$i + 1]
End Loop
If [$error ≠ 0]
    Show Custom Dialog ["Error"; "An error occurred: " & $error]
Else
    Show Custom Dialog ["Success"; "1000 records were successfully created."]
End If

Evaluation: ChatGPT’s script effectively uses error capture to prevent default error dialogs from interrupting the script. The script correctly handles errors within the loop, ensuring robust execution and accurate error feedback. This response demonstrates a high level of understanding in both FileMaker scripting and error management.

Quality Score: 95/100

• Reasoning: Demonstrates excellent understanding of FileMaker scripting, including robust error handling and practical script structure. Minor deductions for potential improvements in dynamic field value setting or user interaction optimization.

2. GPT-3.5 (Free Version)

Script:

Set Error Capture [On]
Loop
    Set Variable [$i; Value:$i + 1]
    Exit Loop If [$i > 1000]
    New Record/Request
    Set Field [myTable::MyField; "YourFieldValue"]
    Commit Records/Requests [No dialog]
End Loop
If [Get(LastError) = 0]
    Show Custom Dialog ["Success"; "1000 records have been created successfully."]
Else
    Show Custom Dialog ["Error"; "An error occurred: " & Get(LastError)]
End If

Evaluation: The script from GPT-3.5 is correct and functional but lacks immediate error handling after each critical operation, potentially missing specific errors during record creation and field setting.

Quality Score: 85/100

• Reasoning: A solid and functional script with correct error handling at the end of the operations. The score is slightly lower due to less immediate error handling compared to GPT-4, which could miss specific issues during execution.

3. Meta.ai

Script (Incorrect Approach):

Tell Application "FileMaker Pro"
    Activate
    Set Variable [$counter; Value:0]
    Set Variable [$error; Value:""]
    Loop
        Exit Loop If [$counter ≥ 1000]
        New Record/Request
        Set Field [myTable::MyField; "Record " & $counter]
        Commit Record/Request
        Set Variable [$counter; Value:$counter + 1]
    End Loop
    If [Get(LastError) = 0]
        Show Custom Dialog ["Success!"; "1000 records created successfully."]
    Else
        Set Variable [$error; Value:Get(Error Message)]
        Show Custom Dialog ["Error"; "An error occurred: " & $error]
    End Tell

Evaluation: This script mistakenly uses AppleScript syntax, reflecting a fundamental misunderstanding of FileMaker scripting environments. FileMaker scripts do not support Tell Application syntax, making this response incorrect for the given task.

Quality Score: 10/100

• Reasoning: Incorrectly uses AppleScript syntax mixed with FileMaker scripting commands, indicating a fundamental misunderstanding of the task requirements and environment.

4. Mistral.ai

Script:

Set Variable [$counter; Value: 1]
Set Variable [$error; Value: ""]
Loop
    Exit Loop If [$counter > 1000]
    New Record/Request
    Set Field [myTable::MyField; "Record " & $counter]
    Commit Records/Requests []
    Set Variable [$counter; Value:$counter + 1]
End Loop
If [$error ≠ ""]
    Show Custom Dialog ["Error"; $error]
Else
    Show Custom Dialog ["Success"; "1000 records have been created."]
End If

Evaluation: Mistral AI’s script does not activate error capturing and potentially leaves the system vulnerable to error dialog interruptions, which can disrupt the script’s execution flow.

Quality Score: 70/100

• Reasoning: Correct script format but lacks sophisticated error handling mechanisms like immediate checking after each operation. Misses best practices that could prevent disruptions in script execution.

5. Pi.ai

Script:

Allow User Abort [ Off ]
Go to Layout [ “myTable” (myTable) ]
Loop
    Set Variable [ $counter; Value:$counter + 1 ]
    New Record/Request
    Set Field [ myTable::MyField; $counter ] Exit Loop If [ $counter = 1000 ]
End Loop
If [ Get (LastError) = 0 ]
    Show Custom Dialog [ Message: "Success! 1000 records have been created in myTable::MyField."; Buttons: “OK” ]
Else
    Show Custom Dialog [ Message: "Error creating records: " & Get (LastError); Buttons: “OK” ]
End If

Evaluation: While Pi.ai correctly sets up a loop for record creation, its decision to disable user aborts without setting error capture could lead to situations where the user cannot interrupt a malfunctioning script, potentially causing application hang-ups.

Quality Score: 50/100

• Reasoning: While the script setup for record creation is logical, the failure to include error capture and the potential to hang the system with “Allow User Abort [Off]” without corresponding error management lowers its score.

6. Perplexity.ai

Script (Faulty):

Set Variable [$counter; Value: 1]
Loop
    Set Field [myTable::MyField; "Record " & $counter]
    Commit Records/Requests [No dialog]
    Set Variable [$counter; Value: $counter + 1]
    Exit Loop If [$counter > 1000]
End Loop
If [Get ( LastError ) = 0]
    Show Custom Dialog ["Success"; "1000 records created successfully."]
Else
    Show Custom Dialog ["Error"; Get ( LastError )]
End If

Evaluation: This response fails to include a New Record/Request step, which is essential for actually creating records in the database, rendering the script ineffective for its intended purpose.

Quality Score: 15/100

• Reasoning: Critical failure by not including the New Record/Request step, making the script ineffective for the intended task. This shows a significant oversight in understanding basic FileMaker operations.

7. Copilot Free

Script:

Go to Layout ["myTable"]
Set Variable [$i; Value:1]
Loop
    New Record/Request
    Set Field [myTable::MyField; "Your desired value here"]
    If [Get (LastError) ≠ 0]
        Show Custom Dialog ["Error"; "An error occurred while creating records."]
        Exit Script
    End If
    Go to Record/Request/Page [Next; Exit after last]
    Set Variable [$i; Value:$i + 1]
    Exit Loop If [$i > 1000]
End Loop
Show Custom Dialog ["Success"; "1000 records created successfully!"]

Evaluation: This script appropriately navigates to the necessary layout and checks for errors right after each critical operation. However, the loop’s exit conditions and error handling could potentially lead to excessive record navigation, impacting performance. The script lacks a method to set error capturing, which might allow system error dialogs to disrupt the script execution.

Quality Score: 75/100

• Reasoning: Properly navigates to the layout and checks errors efficiently. However, the script could be improved in terms of managing potential performance issues due to unnecessary record navigation.

8. Blackbox.ai

Script:

Set Error Capture [On]
Loop
    Set Variable [$i; Value:1]
    Loop
        New Record/Request
        Set Field [myTable::MyField; "Record " & $i]
        Commit Records/Requests
        Set Variable [$i; Value:$i + 1]
        Exit Loop If [$i > 1000]
    End Loop
End Loop
If [Get(LastError) = 0]
    Show Custom Dialog ["Success"; "1000 records created successfully in myTable::MyField"]
Else
    Show Custom Dialog ["Error"; "Error creating records: " & Get(LastError)]
End If

Evaluation: The nested loops in this script seem redundant and might be a mistake, leading to incorrect script operation. The error handling is appropriate, but the loop structure needs correction.

Quality Score: 40/100

• Reasoning: While the script includes the fundamental operations needed to manipulate records in FileMaker, the redundant loops significantly detract from its efficiency and clarity. Such structural issues are not just minor nuisances but can lead to more severe logical errors, potentially causing the script to behave unpredictably or inefficiently, especially when scaled up.

9. Gemini Free (Multiple Drafts)

Example Draft Script:

Set Variable [$counter; Value: 0]
Loop
    Exit Loop If [$counter >= 1000]
    New Record/Request
    Set Field [myTable::MyField; "Record " & $counter]
    Set Variable [$counter; Value: $counter + 1]
End Loop
If [Get(LastError)]
    Show Custom Dialog ["Error"; "An error occurred while creating records: " & Get(LastError)]
Else
    Show Custom Dialog ["Success"; "Successfully created 1000 records!"]
End If

Evaluation: Gemini free versions offer basic script formats focusing on the core task without layout navigation or advanced error handling strategies. The script setup is simple and direct, suitable for uncomplicated applications but lacking in detailed error management.

Quality Score: 60/100

Reasoning: Offers a basic yet direct approach, suitable for straightforward tasks but lacking advanced features such as detailed error feedback or layout management, which limits its practical use.

10. Gemini Pro (you need a Google Cloud Account)

Script:

# Set the number of records to create
Set Variable [ $recordCount; Value: 1000 ]

# Loop to create records
Loop
  # Exit loop if desired number of records reached
  Exit Loop If [ Get ( RecordNumber ) ≥ $recordCount ]
  
  # Create a new record
  New Record/Request
  
  # Set field value (replace with your actual field name)
  Set Field [ myTable::MyField; "Record " & Get ( RecordNumber ) ]
  
  # Commit record to save changes
  Commit Records/Requests [ No dialog ]
End Loop

# Check for errors
If [ Get ( LastError ) = 0 ]
  # No error, display success dialog
  Show Custom Dialog [ 
    Title: "Success"; 
    Message: "1000 records created successfully!"; 
    Buttons: "OK" 
  ]
Else
  # Error occurred, display error message
  Show Custom Dialog [ 
    Title: "Error"; 
    Message: Get ( LastError ) & ": " & Get ( LastExternalErrorDetail ); 
    Buttons: "OK" 
  ]
End If

Evaluation: Get(LastExternalErrorDetail), is not a valid function in FileMaker, indicating a lack of precision in the script’s compatibility with FileMaker’s scripting language. This detail could mislead users or cause script errors if not caught during review. Such an error is crucial because it could prevent the script from running correctly, affecting the script’s reliability and robustness.

Quality Score: 70/100

• Reasoning: Shows a more advanced scripting approach with good use of record handling and error management. However, The script is largely functional and shows an advanced approach to handling record creation loops. However, the error with LastExternalErrorDetail shows a need for more rigorous validation of platform-specific functions, which is crucial for ensuring script reliability in a production environment.

11. Claude 3

Script:

Set Variable [$numRecords; Value: 0]
Set Variable [$error; Value: ""]
Enter Browse Mode [Pause: Off]
Go to Layout ["myTable"]
Loop
    Set Variable [$numRecords; Value: $numRecords + 1]
    New Record/Request
    Set Field [myTable::MyField; "Record #" & $numRecords]
    If [Get(LastError) ≠ 0]
        Set Variable [$error; Value: Get(LastErrorCode) & ": " & Get(LastErrorMessage)]
        Exit Loop [Next Entry]
    End If
    If [$numRecords = 1000]
        Exit Loop [Next Entry]
    End If
End Loop
If [$error ≠ ""]
    Show Custom Dialog ["An error occurred"; $error]
Else
    Show Custom Dialog ["Success"; "1000 records were created successfully."]
End If

Evaluation: Claude 3 correctly navigates to the appropriate layout and checks errors efficiently after each operation, ensuring a robust error handling mechanism is in place. This response demonstrates a sophisticated approach to scripting by also integrating detailed error information, making debugging easier.

Quality Score: 90/100

• Reasoning: Excellent use of layout navigation and detailed error diagnostics. Scores highly for robust error handling and practical script structuring, making it a reliable choice for complex scripting needs.

Conclusion

The comparative analysis shows a wide range of capabilities and understanding among different AI tools. While some, like ChatGPT and Claude 3, demonstrate a strong grasp of FileMaker scripting and best practices, others exhibit fundamental errors that could lead to inefficient or incorrect script execution. Users of these AI-generated scripts must critically evaluate and possibly refine them before deployment to ensure they meet the specific needs and standards of their FileMaker applications.