Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Ozle]

Having an argument with a developer at work over the chance of an error happening.

We run ~200 payrolls a month.
Each is assigned the next sequential Batch number when it runs of 001 to 999 (other things use these batch numbers as well so its not a straight pattern, but there are less than 999 jobs run a month so each number will only be used once a month, so for the purpose of this test I am assume a randomly assigned number each month)

If a payroll picks up the same number as before, it errors (well it actually dissapears the previous file, but for the purpose of this case lets just say error)

Now our developer is arguing its not a problem because it only has a 1 in 999 chance of picking the same number after one month, and a 2 in 999 chance the second month etc..

Im trying to point out that because we run 200 payrolls the chances of us encountering on ANY payroll this error is not the 1 in a 1000 months he keeps saying.


I tried to explain it with coins to keep it simple
If i flip Coin A and Coin B
Coin A, comes up Heads
Coin B comes up Tails

If I flip Coin A again
Chance of coming up Heads   1/2

If I flip Coin B again
Chances of coming up Tails  1/2

If I flip them both at the same time
Chances of A coming up Heads OR  B coming up tails IS NOT 1/2
  its 3/4
 (Options are
AA
AB
BA
BB)

But he is not having it.

Obviously the chance of it happening in the first month is 0
And after the 999 months the chances are obviously 100%
What are the chances in the second month?
And the third?

[Watno]

Try telling him this: http://en.wikipedia.org/wiki/Birthday_problem

So the error occurs if any previous number is matched?

[Ozle]

Try telling him this: http://en.wikipedia.org/wiki/Birthday_problem

So the error occurs if any previous number is matched?

Yes, a previous number matched by the same payroll.

[Watno]

http://en.wikipedia.org/wiki/Random_permutation_statistics#Number_of_permutations_that_are_derangements

I knew i had heard that before: For the 2nd month, the chance that the error occurs is already >63% (It converges to 1-1/e when the number of people goes up, and it does so really quickly).

[Warfreak2]

Yes, this is a birthday problem. If there were 365 numbers then after 23 payrolls an error is more likely than not to have occurred. For 1000 numbers, 38 payrolls makes an error more than 50% likely, and if your tolerance for errors is lower, say 5%, 11 payrolls is enough. See also: http://lazycackle.com/Probability_of_repeated_event_online_calculator__birthday_problem_.html

[Warfreak2]

I don't know where uou got 1/e from, the probability of an error clearly converges to 1 as the number of payrolls increases.

[Watno]

Also see this plot and the table of values below it.

However I just noticed taht doesn't account for the fact that it's not always the same set of 200 numbers we're talking about.

@Warfreak: The 1-1/e was for a single month.

[Watno]

I think youre misunderstanding the problem Warfreak. It's not actually the same as the birthday problem. Two people can't get the same number in the same month, but the problem occurs if someone get a number hes had before.

[Warfreak2]

It is precisely the birthday problem, just for each person rather than each month. After 1001 months the probability is certainly not 1 - 1/e.

[Watno]

Ok, you can indeed look at it from this direction as well.

Forgot to link my plot earlier: http://en.wikipedia.org/wiki/File:N!_v_!n.svg

BTW my proposal would be to let him run a simulation on it: Preferably make a bet on whos right before.

[Warfreak2]

The simple solution, if your developer is innumerate, especially if it's his responsibility to not create errors, is to just wait a few months and he'll see the problem himself. It sounds like he isn't much of a developer anyway, if he decides to use numbers up to 999 instead of, you know, timestamps and user ids.

[HiveMindEmulator]

The problem as posed is the same as the birthday problem with payees playing the role of days and batch numbers playing the role of people.

But the model of random number assignment is questionable, since you are actually cycling through the numbers, not using a RNG. If the number of payees, N, is fixed, there won't be a repeat until the Mth month where M=LCD(N,999), at which point everyone will repeat.

[Ozle]

The simple solution, if your developer is innumerate, especially if it's his responsibility to not create errors, is to just wait a few months and he'll see the problem himself. It sounds like he isn't much of a developer anyway, if he decides to use numbers up to 999 instead of, you know, timestamps and user ids.

i'd love to do that....unfortunately my staff would be the ones to deal with the consequences of his errors...and the errors mean we will lose payroll data for clients (Not something I want to happen)

I'm pretty sure I can convince the bosses that it is indeed an important issue (Hell even at 1/1000 its an important issue)

But just for my own piece of mind I wanted some hard numbers

[Warfreak2]

Considering permutations, you'll get collision probabilities for adjacent months, i suppose. You should take your 1 - 1/e and multiply by 200/999 though, as collisions only matter in 200 of the positions.

[Watno]

The calculator you posted seems to be wrong to me, Warfreak.
It outputs probabilities >0 for making the same choice more than once when only making 1 choice.

[Warfreak2]

There may be unequal numbers generated each month due to staff joining and leaving, or pay slips being corrected for mistakes. Anyway your developer is incompetent, files should have unique, useful names such as 2013-12-10_e42_John_Smith where the payroll for John Smith, employee id e42, has a payslip generated on Dec 10th 2013.

[Warfreak2]

Ok, was just the first google result i found, probably just that edge case, but you can calculate it yourself by doing 1 - (999Pr/999^r) where nPr is the permutation symbol.

[Ozle]

There may be unequal numbers generated each month due to staff joining and leaving, or pay slips being corrected for mistakes. Anyway your developer is incompetent, files should have unique, useful names such as 2013-12-10_e42_John_Smith where the payroll for John Smith, employee id e42, has a payslip generated on Dec 10th 2013.

Yes, this will be my proposed solution. Add the Payroll number to the Batch Number
(these are whole payrolls rather than just employees)

To be fair to him, I dont think he designed the system in the first place, its been running for about 20 years (its an old COBOL mainframe), but only now have we started using it for this new purpose

[Watno]

So I think by Inclusion/Exclusion the exact value we're looking for is
Sum(k=1 to n) ((-1)^k+1 (n choose k) p^k),
where n is the number of employees,  p is the probability of a single employee getting the same number multiple times, which is p=1-(1000 choose m)/1000^m, where m  is the number of months.

But probably there's a mistake in there somewhere.

[Watno]

This, basically:
(I should really learn LATEX, btw)

[Watno]

Nope, that value is actually way too high. It doesn't take into account that the chance of getting a repeat number is increased if someone else does.

[DG]

When you say the payroll goes into error when it picks up the same number as before, do you mean
- the number has been used previously in this month for any employee
- the number has been used in the previous month (for anything)
- the number has been used before (ever)
- the number has been assigned to that person before (ever)
- the number is the same for two successive records read from file
- or anything else

If I remember COBOL data systems correctly, the error might come from trying to write the data record to file using this payroll number as a unique key. If the payroll number is not unique the record will be rejected and not added to file. A simple solution may be to trap this error and immediately assign a new payroll number (You would obviously need to do this before the invalid payroll number is used elsewhere in the system). The preferred solution is of course to append the date and employee code to the payroll record but COBOL does have fixed length data assignment and that would make this change more work than it would be in modern computer languages.

[Ozle]

Just tried to clarify that part, there are 200 separate payrolls, not payees on a payroll. It is whole files not individual transactions within a file.

If an job runs, and the same payroll is assigned to the same batch number as it had been given in a previous month, it causes the system to overwrite the previous file (Henceforth known as 'An Error)

For example
Payroll FM1111 runs in Jan and is given batch number 789, creates a file named FM1111789
It runs again in Feb and is given batch number 845, creates a file named FM111845
If it runs in March and is given batch number 789 it will overwrite Jans file and FM1111789 will only contain March data.


Its one of those problems where it worked fine for when it was originally invented, but nobody planned far enough ahead.

[Thisisnotasmile]

I note Ozle is +1ing every post in this thread.

Just saying...

[Thisisnotasmile]

Whoops I seem to have double posted.