10-12-23 Experiments

Verify: Does LLM make more trivial mistakes when translating?

• Naive code translate attempt(Chat-GPT)
  • Translating 50 APPS python solutions to rust
  • Result:
    • compile@1: 62.5%, compile@any: 92%
    • pass@1: 42.6%, pass@any: 70%
• Code translation with Chat-GPT code summary attempt(Chat-GPT)
  • Translating 50 APPS python solutions to rust
  • Provided with summary of source code by Chat-GPT
  • Result:
    • compile@1: 44.7%, compile@any: 100%
    • pass@1: 28.3%, pass@any:70%
• Directly solve APPS attempt (Chat-GPT)
  • Giving Chat-GPT APPS task prompts directly
  • Result:
    • compile@1: 81.8%, compile@any: 100%
    • pass@1: 18.6%, pass@any:56%

Appendix

Translation with summary prompt example

Translate the following python program using the given python code and summary of program behavior

SUMMARY:

This code prompts the user to enter an integer and then enters a loop based on that integer. Within the loop, the code prompts the user to enter another integer and creates empty lists and sets to store various values.

The code then enters another loop based on the second integer entered. Within this loop, it prompts the user to enter a string and appends it to the 'mass' list. It also checks the first and last character of the string to determine if it matches certain conditions. Based on the conditions, it adds the string to specific sets and lists, and increments different counters.

After the second loop, the code checks if certain conditions are met and either prints -1 and continues to the next iteration of the loop, or proceeds to calculate and print a value based on the counters and sets/lists created earlier. The code then enters another loop using the calculated value and performs calculations based on the contents of the sets/lists to create another list containing certain values. Finally, it prints the contents of the list.

CODE:

for _ in range(int(input())):
    n = int(input())
    mass = []
    zo = 0
    oz = 0
    zz = 0
    oo = 0
    ozs = []
    zos = []
    ozss = set()
    zoss = set()
    for j in range(n):
        k = input()
        mass.append(k)
        if k[0] == '0' and k[-1] == '1':
            zoss.add(k)
            zos.append(j + 1)
            zo += 1
        elif k[0] == '1' and k[-1] == '0':
            ozss.add(k)
            ozs.append(j + 1)
            oz += 1
        elif k[0] == '0' and k[-1] == '0':
            zz += 1
        else:
            oo += 1
    if zz and oo and not oz and not zo:
        print(-1)
        continue
    else:
        if zo > oz:
            print((zo - oz) // 2)
            ans = []
            need = (zo - oz) // 2
            i = 0
            while need:
                zzz = mass[zos[i] - 1][len(mass[zos[i] - 1]) - 1:: -1]
                if zzz not in ozss:
                    ans.append(zos[i])
                    need -= 1
                i += 1
            print(*ans)
        else:
            print((oz - zo) // 2)
            ans = []
            need = (oz - zo) // 2
            i = 0
            while need:
                zzz = mass[ozs[i] - 1][len(mass[ozs[i] - 1]) - 1:: -1]
                if zzz not in zoss:
                    ans.append(ozs[i])
                    need -= 1
                i += 1
            print(*ans)