We examine how the average of double-winding Wilson loops depends on the number of color N in the SU(N) Yang-Mills theory. In the case where the two loops C1 and C2 are identical, we derive the exact operator relation which relates the doublewinding Wilson loop operator in the fundamental representation to that in the higher dimensional representations depending on N. By taking the average of the relation, we find that the difference-of-areas law for the area law falloff recently claimed for N = 2 is excluded for N ⩾ 3, provided that the string tension obeys the Casimir scaling for the higher representations. In the case where the two loops are distinct, we argue that the area law follows a novel law (N − 3)A1/(N − 1) + A2 with A1 and A2(A1 < A2) being the minimal areas spanned respectively by the loops C1 and C2, which is neither sum-ofareas (A1 + A2) nor difference-of-areas (A2 − A1) law when (N ⩾ 3). Indeed, this behavior can be confirmed in the two-dimensional SU(N) Yang-Mills theory exactly.